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ABBREVIATIONS AND ACRONYMS 


Abbreviation Full name 


CARICC The Central Asian Regional 
Information & Coordination 
Centre for Combating Illicit 
Trafficking of Narcotic Drugs, 
Psychotropic Substances & their 


Precursors 
CLP Core Learning Partner 
INL Bureau of International 


Narcotics and Law Enforcement 
Affairs (US State Department) 


IT Information Technology 

LEA Law Enforcement Agency 

ROCA Regional Office for Central Asia 

SDG Sustainable Development Goal 

UNODC United Nations Office on Drugs 
and Crime 

UN United Nations 


MANAGEMENT RESPONSE 


Recommendation 


The main features of the F23 project approach (in particular: 
software provision, appropriate training, support for 
networking, longer-term partnership) should be integrated 
into future similar projects in the region and the new 
Programme for Central Asia. (Project Management, ROCA, 
and Regional Section for West and Central Asia). 
Recommendation is directed to the UNODC Coordinator for 
the Programme for Central Asia and staff responsible for the 
implementation of the successor activities to F23!. 


In follow-on activities as part of the new Programme, 
increase the use of 1) pairings between regional or 
international experts and analysts, 2) trainer-training, and 3) 
training courses held on the premises of beneficiary agencies. 


Identify relevant analyst skills and equipment specifications 
specified as minimum requirements. 

Link training and equipment provision to these identified 
standards, with better target-setting, evaluation and scoring 
for training, with the option to link to national training 
curricula and academies. Support individual countries to 
implement their own training to these standards, and also 
define mid-level and advanced areas of work where outside 
help is needed. 


Develop a concise and realistic logical framework for future 
follow-on activities which interfaces with the minimum 
standards, above, and satisfies usual standards of evaluability 
and logical consistency, formulated ina SMART way and 
used for real-time project monitoring. 


Strengthen procedures for identifying and addressing 
emerging threats in real time, for example by facilitating 
more rapid input of expertise from outside the region and 
more rapid transfer within it. 


Highlight potential human rights implications of the use of 
software and techniques, and help agencies work together on 
possible solutions. 


1 All the recommendations are directed to this same recipient 
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Management Response 
(accepted/partially 
accepted/rejected) 


Accepted 


Accepted 


Accepted 


Partially accepted 


Partially accepted 


Rejected 


Ensure the inclusion of gender equality and collect gender- 
disaggregated data for training and expert meetings. Model 
an inclusive approach in project implementation with key 
individuals / champions who could act as change agents. 


Vil 


Partially accepted 


EXECUTIVE SUMMARY 


Background 


The annual increase in the flow of drugs, mainly opiates from Afghanistan, has become a significant 
external threat for the Central Asian region. Since a key issue in the fight against illicit drug 
trafficking is the collection, analysis and exchange of information carried out by analytical units in 
the law enforcement agencies of the region, the United Nations Office on Drugs and Crime 
(UNODC) project RER/F23 (F23), entitled “Drug law enforcement systems for criminal 
intelligence collection, analysis and exchange”, was designed to fill this gap. The project originally 
began in 2002 and ended on 12 November 2017. The objective of the project was set as “a coherent 
and comprehensive system for intelligence, and information collection, analysis and exchange is 
established”. The project installed modern intelligence systems and i2 analysis software (“iBase” 
and “Analyst’s Notebook”) in 18 counter narcotic agencies in the project area. It was funded by 
Austria, Italy, Canada, France, Norway, Turkey, United Kingdom and United States of America 
through their respective agencies and by UNDP, with a total overall budget of USD 8,016,445 and 
implemented by UNODC Regional Office for Central Asia (ROCA). 


The outcomes of the project are specified somewhat differently in the original project document 
and revisions’, but taking all documentation together, the project aimed at a minimum, within at 
least one agency within each participating country, with respect to trafficking of narcotics, to 
increase/enhance 1) information gathering and 2) analysis, and 3) this was to be increasingly 
intelligence-led; and in addition it also intended to achieve: 1) progress on these dimensions in 
more than one agency in each country; 2) application a) in particular to transnational narcotics 
trafficking and b) more generally to other organized crimes; 3) increased exchange of intelligence 
between agencies within one country; and 4) increased exchange of intelligence between countries. 


The evaluation followed the Organization for Economic Co-operation and Development - 
Development Assistance Committee (OECD — DAC) criteria assessing relevance, efficiency, 
effectiveness, impact and sustainability, as well as partnerships and cooperation, human rights and 
gender equality, and the implementation of recommendations from the mid-term Independent 
Project Evaluation in 2012, deriving lessons learned, best practices and recommendations for future 
project interventions. Relevant lessons and recommendations for the new UNODC Regional 
Programme for Central Asian States 2015-2019 were also sought. The evaluation was undertaken 
by means of a mixed-methods approach with a gender-responsive evaluation methodology in line 
with United Nations Evaluation Group (UNEG) and UNODC Norms and Standards. 


The evaluation time scope was | January 2012 to 2 December 2017 (end of the evaluation field 
mission) and the geographical coverage was Kazakhstan, Kyrgyzstan, Tajikistan and Uzbekistan, 


2 It proved difficult to isolate from the project documentation precisely what were the intended outcomes, so for 
the purposes of the evaluation, ,,essential“ and ,,additional“ success dimensions were defined. The former 
three are considered to be essential minimum outcomes, and the latter four are ,,nice to have“. 
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the locations within which the project was actually implemented. The evaluation was carried out 
by a team of two external independent evaluators, one lead evaluator with experience in evaluating 
technical assistance projects in over 30 countries over the last 20 years, and one team member, 
expert in the area of law enforcement in Central Asia. The evaluation methodology considered 
primary and secondary data sources ensuring triangulation of findings, further promoting the 
participation of stakeholders throughout the evaluation process, including a desk review and field 
mission with semi-structured interviews and site visits including site observation to beneficiary 
agencies, from 20 November to 2 December 2017. 


Main findings 
Relevance 


The project was highly relevant to beneficiary needs and priorities. Improving the effectiveness of 
drug law enforcement is a priority and an urgent and continuing need. The Outcomes were highly 
relevant to the project aims and the Outputs were appropriate to achieve the Outcomes. The long- 
term approach and partnership was an appropriate way to provide this kind of support in this region; 
the duration was however unusually long considering that essentially the same project model was 
used throughout. 


Efficiency 


Achievement of outputs approximately followed the project plans, although already in 2012 the 
geographical scope of the project had to be restricted to just the four evaluated countries, 
Kazakhstan, Kyrgyzstan, Tajikistan and Uzbekistan, due mainly to lack of funding. Project 
activities were cost-effective ways of achieving outcomes. 


Effectiveness 


The project reached the essential standards of progress implied in the logical framework and also 
showed additional successes on a number of other areas, in particular having effects within multiple 
agencies within individual countries and facilitating networking between them and with foreign 
agencies — as evidenced by inspection of project documentation and interviews with internal and 
external respondents. 


The F23 training approach reached a certain level of maturity and delivered good results in all three 
outcome areas, especially the first two (1. Specialized law enforcement personnel use 
enhanced management of information-gathering and exchange systems; 2. Law enforcement 
personnel use enhanced analytical capacity in operational work). As evident from interviews, 
analysts were more in a position to use the skills and capabilities they had acquired through the 
project in their daily work. The approach propagated by F23 became a central part of law 
enforcement operations in some if not all agencies. “Rotation” of trained staff into other positions 
did happen to a considerable extent but was not only a disadvantage because trained staff could use 
their new skills in new areas, and the agencies as a whole could benefit from wider understanding 
of the ways of working introduced and strengthened by F23. 


The project sponsored bilateral links indirectly but not directly: it was indirectly conducive to 
cooperation between relevant agencies in individual countries, and its support for The Central Asian 
Regional Information and Coordination Centre for Combating Illicit Trafficking of Narcotic Drugs, 
Psychotropic Substances and their Precursors (CARICC) contributed to very important regional 
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networking. Regional level assistance was carried out as planned. In-house training was restricted 
to specific needs on particular occasions, in particular when software was installed, though it was 
highly valued. 


The project management succeeded in a demanding task with relatively sparse resources; however, 
the project monitoring system was not really fit for purpose. The logframe items, indicators, 
baselines and targets often did not match up with one another and many were not SMART3. 


Impact 


The project made plausible contributions to the Sustainable Development Goals (SDGs) and target 
16.A (“Strengthen relevant national institutions, including through international cooperation, for 
building capacity at all levels, in particular in developing countries, to prevent violence and combat 
terrorism and crime”) in particular. There was no evidence of a substantial impact on gender- 
specific SDGs. 


The project contributed to a larger number of more evidence-based prosecutions, which was a 
significant achievement, and there were some positive results in neighboring thematic areas which 
were not directly intended — for example, F23 support to CARRIC contributed indirectly to the staff 
of that organisation holding training courses outside the region; and F23 contributed to the 
Afghanistan Kyrgyzstan Tajikistan “Initiative AKT”4. 


Sustainability 


It is highly likely that most agencies would continue with the use of i2 and intelligence-led policing 
within each country even in the absence of similar support; however, regional cooperation, mutual 
support in training etc, would be less likely to sustain. The training model is on the verge of 
becoming sustainable. 


Human Rights and Gender Equality 


Neither gender equality issues nor human rights issues were explicitly included in the project 
design, training materials, or in the way the F23 approach was to be implemented. In interview, all 
three groups of respondents said that the project most likely had a very positive impact on human 
rights by increasing the role, availability and value of evidence in the anti-narcotics process, 
supporting the rule of law. The evaluation showed that the human rights implications (data 
protection etc) need to be made clearer in the training courses in order to increase respect for human 
rights in the operations of F23-related activities. Around 10% of trainees were women, a ratio which 
is typical for law enforcement in Central Asia, though there was some notable improvement in 
2017. 


The evaluation also addressed gender issues, looking in particular at the gender ratio amongst 
analysts and on training courses. Inspection of project records shows that the gender balance 


3 SMART is a much-used acronym which originally meant “Specific — target a specific area for improvement; 
Measurable — quantify or at least suggest an indicator of progress; Assignable — specify who will do it; 
Realistic — state what results can realistically be achieved, given available resources; Time-related — specify 
when the result(s) can be achieved.” as attributed to Doran, G. T. (1981). "There's a S.M.A.R.T. way to 
write management's goals and objectives". Management Review. AMA FORUM. 70 (11): 35-36. 

4 “The Afghanistan-Kyrgyzstan-Tajikistan (AKT) Initiative is aimed at strengthening cross-border cooperation in 
law enforcement and legal matters between these countries at the starting point of the Northern Route for 
trafficking narcotics from Afghanistan.” Source: 
https://www.unodc.org/documents/rpanc/RP_Success_Stories_distribution.pdf 


amongst analysts is still quite poor. Gender-specific records were not held for the training courses, 
but closer inspection of records shows that there was a relatively low percentage of women trained, 
roughly corresponding to the low percentage amongst analysts in general. 


Main conclusions 


The project continued to be highly relevant to the changing situation on narcotics trafficking in 
Central Asia, from the perspective of both national governments and the international community. 
F23 fulfilled a need for outside expertise, respecting the individual situation and needs of each 
beneficiary agency. 


The 2012-17 phase of the project was able to build on and expand the groundwork done in the first 
phase. It did not merely maintain a minimum level of capacity in each agency but on the contrary 
began to reap the benefits in terms of actual use of that capacity to facilitate evidence-based 
prosecutions. 


With respect to the minimum achievement criteria implied in the logical framework the project was 
effective, and also added value, such as including several agencies within each country and helping 
to improve interaction between them. To a lesser extent it managed to contribute to cross-border 
exchange of intelligence but there would still be a lot of work to be done in that regard, some of 
which is beyond the remit of F23. 


Training components reached a certain level of maturity and national agencies were beginning to 
take over responsibility for them, with some material being included in basic material for Police 
Academy trainings. 


While some of the recommendations from the mid-term evaluation in 2012 were taken into account, 
more could have been done to revisit them. In particular, in-house training, train-the-trainer and 
peer-to-peer training were efficient, effective and valued by partners but due to funding restrictions 
were not used as much as recommended by the previous evaluation. 


The project monitoring system and logical framework as formulated in the project document and 
revisions were perhaps its weakest point. They were logically inconsistent and lacked SMART 
formulation. There was no systematic monitoring of e.g. staff scores on a standardised skills test; 
prosecutions brought, etc. 


Key recommendation 


Revisiting recommendations from Mid-Term Evaluation Report 

In follow-on activities as part of the new Programme, increase the use of 1) pairings between 
regional or international experts and analysts, 2) trainer-training, and 3) training courses held on 
the premises of beneficiary agencies. 


Sustainability of expertise; benchmarking 
Identify relevant analyst skills and equipment specifications specified as minimum requirements. 


Link training and equipment provision to these identified standards, with better target-setting, 
evaluation and scoring for training, with the option to link to national training curricula and 
academies. Support individual countries to implement their own training to these standards, and 
also define mid-level and advanced areas of work where outside help is needed. 
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Logical Framework and Monitoring 

Develop a concise and realistic logical framework for future follow-on activities which interfaces 
with the minimum standards, above, and satisfies usual standards of evaluability and logical 
consistency, formulated ina SMART way and used for real-time project monitoring. 


Emerging challenges 
Strengthen procedures for identifying and addressing emerging threats in real time, for example by 
facilitating more rapid input of expertise from outside the region and more rapid transfer within it. 


Human rights & gender 
Highlight potential human rights implications of the use of software and techniques, and help 
agencies work together on possible solutions. 


Collect gender-disaggregated data for training and expert meetings. 
Model an inclusive approach in project implementation with key individuals / champions who 
could act as change agents. 


Lessons learned and best practices 


One best practice was the exemplary long-term and individualized approach which accompanied 
national partners in a difficult context over more than one and a half decades. Patient and persistent 
help with applying the project model led to a major shift in the sense that agencies now see the 
benefits themselves and are motivated to continue the approach. 


The support to expert meetings, alongside training and equipment, helped sponsor a core of key 


personnel with advanced level of expertise within the region, mainly but not only based at 
CARICC. 


xii 


SUMMARY MATRIX OF FINDINGS, EVIDENCE 
AND RECOMMENDATIONS 


Findings 


Evidence (sources that 
substantiate findings) 


Recommendations 


Achievement of outputs 
continued to approximately 
follow the project plans. 
Project activities were cost- 
effective ways of achieving 
outcomes. 


The project certainly reached 
the essential standards of 
progress implied in the 
logical framework and also 
showed additional successes 
on a number of other areas. 


Deskwork (project reports, 
project plans), on-site 
inspection, interview, 
inspection of financial 
records, collation and 


analysis of training records. 


The main features of the F23 
project approach (in particular: 
software provision, appropriate 
training, support for 
networking, longer-term 
partnership) should be 
integrated into future similar 
projects in the region and the 
2015-19 Programme for 
Central Asia. 
Recommendation is directed to 
the UNODC Coordinator for 
the Programme for Central 
Asia and staff responsible for 
the implementation of the 
successor activities to F235. 


Much progress was made 
with respect to the 
recommendations from the 
mid-term evaluation 
(awareness-raising for key 
staff, on-the-job and peer-to- 
peer training). However 
some still need continued 
work and are still valid. 


Project records for 
expert/analyst training etc, 
interview with agency 
representatives and project 
staff. 


In follow-on activities as part 
of the new Programme, 
increase the use of 1) pairings 
between regional or 
international experts and 
analysts, 2) trainer-training, 
and 3) training courses held on 
the premises of beneficiary 
agencies. 


There was a lack of concrete 
evidence of training 
effectiveness apart from 
satisfaction surveys, and no 
systematic way to formulate, 
assess or certify required 
skills and knowledge within 


Project record systems, 
interview with agency 
representatives and project 
staff. 


Identify relevant analyst skills 
and equipment specifications 
specified as minimum 
requirements. 


Link training and equipment 
provision to these identified 


5 All the recommendations are directed to this same recipient 
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or between agencies or 
countries. There was a 
continued need to anchor 
expertise within the region. 


standards, with better target- 
setting, evaluation and scoring 
for training, with the option to 
link to national training 
curricula and academies. 
Support individual countries to 
implement their own training 
to these standards, and also 
define mid-level and advanced 
areas of work where outside 
help is needed. 


The project logical 
framework was weak, 
logically inconsistent and 
lacked SMART formulation. 
There was no systematic 
monitoring of e.g. staff 
scores on a standardised 
skills test; prosecutions 
brought, etc. 


Project document and 
revisions, interviews with 
relevant project staff. 


Develop a concise and realistic 
logical framework for future 
follow-on activities which 
interfaces with the minimum 
standards, above, and satisfies 
usual standards of evaluability 
and logical consistency, 
formulated ina SMART way 
and used for real-time project 
monitoring. 


There are many important 
emerging threats such as the 
use of encrypted messaging 
by criminal groups which 
seriously challenged the 
ability of agencies to respond 
to them; the project’s 
procedures for dealing with 
these threats were not as fast 
as agencies desired. 


Interview with all actors and 
especially government 
agencies; government 
documents. 


Strengthen procedures for 
identifying and addressing 
emerging threats in real time, 
for example by facilitating 
more rapid input of expertise 
from outside the region and 
more rapid transfer within it. 


Use of software and 
techniques provided and 
supported by F23 had some 
potential human rights 
implications (data protection 
etc) which were not dealt 
with thoroughly enough in 
the training courses. 


Interview with agency 
representatives, project staff 
and deskwork. 


Highlight potential human 
rights implications of the use 
of software and techniques, 
and help agencies work 
together on possible solutions. 


Gender equality was not 

specifically addressed in 

project design or training 
materials 


Manual collation and 
analysis of training records 
using trainee names. 
Inspection of project records 


Ensure the inclusion of gender 
equality and collect gender- 
disaggregated data for training 
and expert meetings. Model an 
inclusive approach in project 


xiv 


and materials. Observation 
during evaluation visits. 


implementation with key 
individuals / champions who 
could act as change agents. 
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| INTRODUCTION 


Background and context 


Map 1. Map of the project countries* 
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Key to map: yellow background = all countries in UNODC Strategic Region; thick, dark border = countries included in project 
2012-17 and visited for evaluation; thin, pale border = other countries in project 2002-11. 


*Source: Google Maps 


Afghanistan continues to dominate the world production of opiates, see Table 1. In the 1990s, the 
search for new drug markets led to a new drug transit route through the countries of Central Asia 
to the Russian Federation, called the "Northern Route". The Central Asian countries bordering 
Afghanistan are geographically located in a difficult mountainous area, especially Tajikistan. The 
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and exchange 


annual increase in the flow of drugs has become a significant external threat for the Central Asian 


region. 


Table 1. The cultivation of opium poppy and opium production in Afghanistan 2009- 


2017* 
Year 2012 2013 2014 2015 2016 2017 
Hectares 154000 209000 224000 183000 201000 328000 
Opium 3700 5500 6400 3300 4800 9000 
production (in (estimate) 
metric tons) 


*Data from http://drugsmonitoring.unodc-roca.orgé. 


Countries in the region, including Afghanistan and the Russian Federation, agreed on the need to 
cooperate in the fight against drug trafficking. A special role in this is played by international 
organizations designed to assist the law enforcement agencies of the Central Asian countries. Since 
a key issue in the fight against illicit drug trafficking is the collection, analysis and exchange of 
information carried out by analytical units in the law enforcement agencies of the region, the 
UNODC F723 Project works on achieving these goals. 


Overall concept and design of the F23 project 


The UNODC project RER/F23, informally known as “F23”, had the title “Drug law enforcement 
systems for criminal intelligence collection, analysis and exchange’. The project originally began 
in 2002, since when there have been seven project revisions7, (the most recent, June 2012 and July 
2014, were in the period covered by this evaluation). The objective of project F23 in the most recent 
project revision was “a coherent and comprehensive system for intelligence, and information 
collection, analysis and exchange is established’’s. The project originally began in 2002 and ended 
on 12 November 2017. It was funded by Austria, Italy, Canada, France, Norway, Turkey, United 
Kingdom and United States of America through their respective agencies and by UNDP, with a 
total overall budget of USD 8,016,445 and implemented by UNODC Regional Office for Central 
Asia (ROCA). 


The project area of responsibility at project end included the five Central Asian countries 
(Kazakhstan, Kyrgyzstan, Tajikistan, Uzbekistan and Turkmenistan), Afghanistan and Azerbaijan. 
However, Turkmenistan was not actively involved in the project, Azerbaijan and Afghanistan were 
included up until 2012 and were intended to be included also for the phase 2012-17. Armenia and 
Georgia were also intended to be included 2012-17. See Table 2 During the period of 2012-2017, 
direct support was provided just to the first four of these countries, mainly due to lack of funding. 


6 See also https://www.unodc.org/unodc/en/press/releases/2017/November/afghan-opium-production-jumps-to- 
record-level--up-87-per-cent_-survey.html 

7 See Terms of Reference, p. 44 

8 Changes to the project logical framework, including to the Objectives, are outlined on p. 96. 


INTRODUCTION 


The project installed modern intelligence systems and i2 analysis software (“iBase” and “Analyst’s 
Notebook”) in 18 counter narcotic agencies in the project area. Approximately 400 officers have 
been trained in crime analysis techniques at various levels. The F23 “project philosophy” was to 
teach, across different agencies and countries, similar working methodologies and using the same 
analysis software and database structures. This approach was intended to stimulate national and 
regional cooperation between agencies and encourage the sharing of information and analytical 
products between intelligence analysts from participating countries. 


Logical framework 


The three Outcomes of the projects (as further elaborated in Annex VIII) are as follows (slightly 
abbreviated). The evaluation team have added italics to words which are presumably intended to 
express the intended changes, the difference made by the project. 


Specialized law enforcement personnel use enhanced management of information-gathering and 
exchange systems. 


Law enforcement personnel use enhanced analytical capacity in operational work. 
Law enforcement personnel increased use of intelligence-led policing in operational work. 


The formulations of these and their indicators, and the formulations of the Objective, are however 
inconsistent and in general do not define (with sufficient clarity) the baselines, or how progress is 
to be measured, or the desired extent / scale of the project. In particular, it is not clear whether the 
targets refer to one beneficiary agency, or all of them, or to individual countries; which kind of 
crime is to be targeted; what kind of data exchange between agencies is intended — within or 
between countries? 


To explicate more precisely what the project was actually aiming to do in order to be able to 
evaluate its actual achievement against those aims, the evaluation team condensed the logical 
framework into essential and additional success dimensions: 


Essential success dimensions: Within at least one agency within each participating country, with 
respect to trafficking of narcotics, the project must increase/enhance the three Outcomes: 1) 
information gathering and 2) analysis, and 3) this is to be increasingly intelligence-led9. 


There are also additional aims which are implied at various points in the project documentation 
but not so consistently that they can be judged to be binding: 


Additional success dimensions: 1) progress on the essential dimensions in more than one agency 
in each country; 2) application a) in particular to transnational narcotics trafficking and b) more 
generally to other organised crimes; 3) increased exchange of intelligence between agencies within 
one country; 4) increased exchange of intelligence between countries 


9 This means above all that the work of agencies is not only responding to crimes but using intelligence to 
understand and anticipate crime in a pro-active way, also impacting resource allocation. 
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Since targets are not consistently given, any substantial difference made to these dimensions will 
be judged as success. Finally, although a wider geographical scope was indicated initially, the “the 
participating countries” is considered to be limited to the four countries covered in the ToR for the 
evaluation. 


Figure 1. A graphical representation of the project logical framework from the latest 
Revision (7), with adjustments 


Outputs 
Output 1.1 
Necessary training for the use of new software is 
provided 
Output 1.2 
Train the trainer approach to develop a 
self-sustaining competence. 
Outcome 1 
Specialized law enforcement personnel use 
Output 2.1 enhanced management of information gathering: foo cicecsssssessesescsesestscsssssssscseececeneneatateceseess 
Conduct an assessment of analytical and and exchange systems. roject Objective: 
intelligence led policing capacity in respond to the coherent and comprehensive system for : 
existing needs. Outcome 2 telligence, information and information collection, ; 
Law enforcement personnel use enhanced : peek alte Sete Ce El : 
Bice CAR BCH IN ODarcHGnal WGrK : Each project country has at least operational 
Output 2.2 aly! pacity in op! . : analysis unit with functional and comprehensive 
Enhance efforts of the beneficiaries for self-funding : database as a support tool for the ongoing 
or direct outreach to potential donors. op nvestigations. 
Outcome3 i ii(‘i~*C*dSC Mean n eee eee ennen ne nenacce cece ceca tnee ne nnncace cece cece cccccened 
Law enforcement personnel increase use of 
Output 3.1 intelligence-led policing tools in operational work 


Necessary IT equipment (hardware and software) 
provided to meet existing challenges. 


Output 3.2 

Organize regular semi-annual regional meetings 
involving participating countries, with a view to 
increasing knowledge and strengthening LE 
(analytical) cooperation and information exchange. 


*All Outputs potentially contribute to each Outcome; dotted lines to and around Objective show it is defined in terms of 


the Outcomes rather than caused by them 


Purpose and scope of the evaluation 


A final Independent Project Evaluation is mandatory and has to take place prior to the financial 
closure of the project as per UNODC Evaluation Policy. 


The evaluation followed the Organization for Economic Co-operation and Development - 
Development Assistance Committee (OECD — DAC) criteria as well as some additional criteria 
(see methodology section on p. 6), deriving lessons learned, best practices and recommendations 
for future project interventions. As many aspects of the project are to be integrated with other 
projects under the umbrella of the UNODC Programme for Central Asian States 2015-2019 signed 
in May 2015 by the government representatives of Kazakhstan, Kyrgyzstan, Tajikistan, 
Turkmenistan and Uzbekistan, relevant lessons and recommendations for the new UNODC 
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Regional Programme for Central Asian States 2015-2019 were also sought. The evaluation was 
undertaken by means of a mixed-methods approach with a gender-responsive evaluation 
methodology in line with United Nations Evaluation Group (UNEG) and UNODC Norms and 
Standards. The evaluation process is fully independent and is conducted under the guidance and 
quality assurance of the UNODC Independent Evaluation Unit (IEU)!°. 


The evaluation criteria as specified in the ToR (and slightly refined in the inception report) form 
the sub-headings of the Findings sections, pp. 10 ff. 


The purpose of the evaluation is to ensure ownership, result-based orientation, cost-effectiveness 
and quality of the UNODC services. 


The results of this summative final evaluation are intended for use by the UNODC Regional Office 
for Central Asia, project team, beneficiary agencies and donor countries. 


The evaluation time scope is 1 January 2012 to 2 December 2017. Furthermore, the project coverage 
is nine countries, however since 2012 F23, funds were only available (from INL USA) for the 
Central Asia states. Out of five Central Asia states Turkmenistan did not sign the project extension. 
AS a result since 2012 the project was implemented only in these four countries. So the evaluation 
geographical scope is as shown in the shaded column: 


Table 2. The project countries 


UNODC | Included | Intended | Actually Visited in | Included in 
Strategic | in 23 | to be | included in | evaluation | new 
Region | 2002-11 | included | phase 2012- Programme 
in phase | 17: for Central 
2012-17 | Geographical Asia 
scope of 
evaluation 
Uzbekistan x Xx Xx x x Xx 
Kazakhstan x Xx Xx x x Xx 
Kyrgyzstan Xx Xx Xx Xx x Xx 
Tajikistan x x x x x x 
Turkmenistan | X Xx Xx Xx 
Afghanistan | X Xx Xx 
Azerbaijan Xx Xx Xx 


10 http://www.unodc.org/unodc/en/evaluation/index.html 
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Armenia xX »4 


Georgia Xx Xx 


The composition of the evaluation team 


The final Independent Project Evaluation was carried out between October 2017 and January 2018 
by a team of two external independent evaluators: 


e International independent external evaluator (lead evaluator), with experience of 
evaluating technical assistance projects, having conducted evaluations in over 30 countries 
over the last 20 years, with a PhD in research psychology from the University of 
Middlesex. 


e National independent evaluator (team member), expert in the area of law enforcement, with 
in-depth knowledge on the drugs situation in Central Asia as Deputy Director of the Drug 
Control Agency in the Republic of Tajikistan!!. 


Logistical support was provided by Project staff. 


Evaluation methodology 


The evaluation followed the Organization for Economic Co-operation and Development - 
Development Assistance Committee (OECD — DAC) criteria, assessing relevance, efficiency, 
effectiveness, impact and sustainability, as well partnerships and cooperation, human rights and 
gender equality mainstreaming, and the implementation of recommendations from the mid- 
term Independent Project Evaluation in 2012. A set of evaluation questions was agreed in the 
inception phase, organised according to the evaluation criteria. These were taken from the ToR and 
further modified as indicated on pp. 72 ff. Then an “evaluation matrix” was constructed with 
evaluation questions, collection methods and data sources. 


For each evaluation question, a corresponding subsection of the report was written. This subsection 
was written as a triangulation of the different results for each of the methods used for that particular 
question, to strengthen accuracy, robustness and reliability of evaluation results. 


In order to ensure the evaluation offered diverse perspectives a variety of different groups of 
stakeholders was included, as detailed below. The list of stakeholders includes seven Core Learning 
Partners (CLPs); these were a limited number of those deemed as particularly relevant to be 
involved throughout the evaluation process, i.e. tasks such as reviewing and commenting on the 
TOR and the evaluation questions. An initial group of stakeholders, and potential interviewees, 
including CLPs, was defined in the evaluation Terms of Reference and included in the first version 


11 The team member was not directly involved in the F23 project and did not take part in any F23 training: as she 
was working as head of a DCA she could not be direct recipient of F23. But she was involved in managing 
the analytical units also during the cooperation with F23. 


INTRODUCTION 


of the evaluation matrix. On inspection of the matrix it was then decided, in the inception phase, to 
also add interviews with two civil society organisations in order to increase the variety of 
respondent types and to broaden the possibilities for triangulation — see p. 9. 


The evaluation followed a gender-responsive approach. Women are very much in a minority in law 
enforcement in the region, so in mixed groups, female respondents were addressed specifically and 
supportively. The evaluators maintained their questioning even if questions relating to gender were 
given lower priority by male respondents. The evaluation team consisted of one woman and one 
man. On occasion, the female evaluator was able additionally to engage female respondents 
individually on issues like workplace environment, during site visits. 


The evaluation made use of a mix of qualitative and quantitative methods to gather and analyse 
data. The methods used included a desk review of documentation, semi-structured interviews and 
to a lesser extent observation. A limited amount of quantitative data was available in the form of 
data on beneficiary agency personnel participating in project events and also in the form of their 
feedback on those events. 


Interviews were of various types, mainly differentiated according to stakeholder group, as follows: 


e Interviews with UN staff, including UNODC. These took place on UN premises (and in 
one case by telephone) with either one or two respondents. 


e Interviews with beneficiary agencies. These took the form of 1-3 hour visits to the premises 
of the Agency in question. The format varied but always involved at least a central 
interview with 1-5 key respondents. In some agencies it was possible also to talk to 
individual agents, in an individual interview. 


e Interviews with donors and others, with either one or two respondents. These included the 
main donor (INL), Ministries of Foreign Affairs, and civil society organisations. 


Different groups of questions from the overall set of evaluation questions were highlighted as 
particularly relevant for each group of stakeholders and operationalised for that group as indicated 
in the evaluation matrix. Interviewees were asked for their consent to the interview before it began 
and were informed about confidentiality. 


Observation was only conducted at beneficiary agencies. For security reasons it was not possible 
to view agents working with live data. However it was possible to view the working environment, 
the equipment which had been provided alongside equipment from other sources, and to ask agents 
questions about their use of the software and procedures. 


Detailed list of evaluation activities 


A desk review of relevant documents (Annex III), followed by preparation of an Inception Report 
(containing preliminary findings of the desk review, refined evaluation questions, data collection 
instruments, sampling strategy, limitations to the evaluation, and timetable). 


Field mission from 20 November to 2 December 2017 in Central Asia. This included the following 
activities. 
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e Visit to UNODC Regional Office for Central Asia (ROCA) in Tashkent consisting 

of: 

Briefing and individual semi-structured interviews of UNODC staff. 

Visits to Kazakhstan, Kyrgyzstan, Tajikistan and Uzbekistan consisting of: 

Individual semi-structured interview with and briefings by UNODC staff members. 

Briefing by UNODC staff in the UNODC Programme-Offices in Kazakhstan, 

Kyrgyzstan, Tajikistan and UNODC Regional Office for Central Asia, Uzbekistan; 

e Individual semi-structured interviews with project beneficiary agencies and other 
national counterparts of Kazakhstan, Kyrgyzstan, Tajikistan and Uzbekistan; 
Individual semi-structured interviews with donor representatives; 
Site visits and observations to Kazakhstan, Kyrgyzstan, Tajikistan and Uzbekistan 
and other sites for physical inspection and discussions both with beneficiary 
agencies and other people directly or indirectly affected by the project; 

e Site visit and observation to CARICC in Almaty 

e Phone/Skype individual semi-structured interviews with staff at UNODC HQ, 
Vienna, Austria. 

e =©Analysis of all available information ensuring triangulation of sources, methods, data and 
theories. 

e Preparation of the draft evaluation report. 

e Preparation of the final evaluation report and presentation of evaluation findings. 


Sampling 


With reference to Table 2 it can be seen that at country level, the sample is a so-called census, i.e. 
all are covered. But within countries, it was not possible for logistics reasons to visit all the 
agencies; in fact, in the selection made by project management, four agencies were left out 
(covering 8 out of 12). This selection had already been discussed and agreed with the partners 
concerned as part of the evaluation ToR and thus was difficult for the evaluation team to change. 


Figure | gives an overview of the interviewees. In total, 53 stakeholders were interviewed, of whom 
10 were female. Beneficiary agencies were best represented, followed by project management. The 
gender balance was most equal in project management. Women made up about 1/5 of the sample, 
which is much better than the ratio in the agencies overall. 
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Figure 2. Individual respondents interviewed for this independent evaluation 


Number of stakeholders interviewed 


Male m Female 


Grand Total 


Project management 


Other Govt. Agency 


Donor 


Civil Society 


Beneficiary Agency 


Limitations to the evaluation 


There was a very limited number of fieldwork days to cover many sites and a long list of evaluation 
questions. This was partially mitigated by submitting some questions in writing to the project 
management. 


It was difficult to assess the main outcomes without clear definition of outcomes / indicators or 
comprehensive baseline or endline. This was partially mitigated by obtaining full project 
monitoring data from management in addition to the annual reports. 


Some of the topics were sensitive (e.g. organized crime) which may compromise / bias some 
sources & the freedom for Law Enforcement Agency staff to discuss high level or sensitive matters. 
This was partially mitigated by adding interviews with CSOs. Circular questioning was also used 
(e.g. what do you think other countries ... / other agencies ... would answer to this question: ...) 


It was difficult to investigate some kinds of possible negative impact. e.g. whether i2 software was 
being used in ways not consistent with human rights, for example to track opposition groups; 
project management had no record of such cases. Partial mitigation was to ask civil society groups 
about this possibility; however, they were reluctant to discuss this kind of question at all, so no 
conclusions could be drawn. 
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Relevance 


Evaluation question: 


To what extent is / was the project relevant to the respective beneficiaries ’ agencies and 
governments’ needs and priorities? 


To what extent is / was the project relevant to the needs and priorities of the donors and UN? 


To what extent do the objectives, outcomes and outputs respond to present circumstances and 
stakeholder expectations 12? 


Relevance to stakeholders 


Improving drug law enforcement is a kind of activity which one could expect to find within 
UNODC’s contribution to the respective UNDAF frameworks. In fact, it is specifically mentioned 
in the UNDAF for Kazakhstan 2010-15; Output 3.4. However, the topic is not specifically covered 
in the UNDAF plans for Kyrgyzstan 2012-16, Tajikistan 2016-20 or Uzbekistan 2016-20. 


On the whole, the project was highly relevant to the appropriate national strategic plans, see p. 
76 and p. 102. The strategic plan for Kyrgyzstan!? in particular mentions ,,carrying out systematic 
combat against corruption, organized crime and drug trafficking. Establishing incentives for 
legalization of informal (shadow) economy; ... Promptly and decisively respond to challenges and 
threats: international terrorism, religious extremism and drug trafficking“. However it is the 
relevance of the project to Kazakhstan’s priorities is less clear, as the national development plan 
for that country only mentions anti-narcotics in terms of reducing addiction and increasing 
penalties. The other three countries’ plans specifically mention installation of databases and other 
appropriate technology. 


In interview, representatives of the beneficiary agencies emphatically agreed that the project 
addressed important national priorities. Narcotics trafficking is the centre of a nexus of problems 
which present real and growing threats to national development, such as money-laundering and 
terrorism, yet it is estimated" that only a small proportion of narcotics passing the national borders 
is actually seized!5. 


12 In this and subsequent sections of this Findings chapter, some additional questions, mainly arising out of the 
mid-term evaluation, are also subsumed under the main headings and can be found in Annex I 

13 National Sustainable Development Strategy for The Kyrgyz Republic 2013-2017 

14 Source: UNODC, 2008: Illicit Drug Trends in Central Asia 

15 In 2010 UNODC estimated that only between 1% and 10% of opium and heroin transiting Tajikistan were 
seized: 75-80 tons of heroin and 18-20 tons of opium trafficked versus 985 kg of heroin and 744 kg of 
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F23 strategy and activities are also in line with the strategy of the main donor, (Bureau of 
International Narcotics and Law Enforcement Affairs; INL) and with that of the US State 
Department, as part of a broader strategy to strengthen criminal intelligence within law enforcement 
agencies in Central Asia. 


Relevance of objectives, outcomes & outputs 


Interview sources and document review concur that some form of international information sharing 
is absolutely essential for transnational crime prevention. The main project outcomes - analysts 
gathering, using and exchanging intelligence not only reactively but also proactively - are highly 
relevant to achieving the project aims (while the Objective is best understood as a synthesis of the 
Outcomes, see p. 3, the Expected Accomplishment!6 is a useful alternative expression of general 
project aims). While gathering, analysing and exchanging intelligence have always been recognised 
as important, the 3rd Outcome on intelligence-led policing in particular is something which became 
clearer as a key contribution partly through the involvement with the F23 project, and in particular 
through witnessing the capabilities of the 12 software. 


Equally, the project activities and outputs were seen both by beneficiary agencies and by external 
respondents on the whole as relevant ways to achieve these objectives. The i2 software was very 
well suited to the task; there are some alternatives but there were no compelling arguments to make 
what would be a very expensive and time consuming shift to a different system. There are also 
relevant potential applications for the software which the agencies are exploring and for which the 
agencies are looking for assistance. 


Telephony meta data, i.e. who called whom, where and when was the most significant component 
of i2 applications in most agencies. There were some questions as to whether the rise of messaging 
and VoIP applications like WhatsApp, which are much harder to trace in the same way, might 
threaten the whole project approach. However, i2 is in principle agnostic as to the type of data being 
used, and most of the agencies were already integrating other types of data such as financial 
transactions; Indeed, some of the best results were achieved where different types of data were 
combined. In parallel, the accompanying hardware and training support provided by the F23 project 
was necessary to make use of the software adequately. The project approach was flexible with the 
right balance between software, hardware and training. 


The project was viewed by beneficiary agencies and by external respondents as the right kind of 
partner to offer this package of support. 


The project took an unusually long voyage, from 2002 to 2017. This kind of long-term relationship 
was explicitly welcomed by the respondents, even when there were very few staff in place who had 
actually experienced more than a few years of it. It was taken as indicating a long-term commitment 
to supporting the agencies and a sign of patience and the understanding that it can take a long time 
to achieve sustainable change. It also made it easier for the project to identify partners and target 
those who were best placed to be effective. 


opium seized. Source: Opiate Flows Through Northern Afghanistan and Central Asia: A Threat Assessment. 
United Nations Office on Drugs and Crime (2012) 
16 General Assembly resolution 64/179 


11 


The evaluation showed that achieving progress was more difficult in the earlier years of the project. 
It took some years until a “critical mass” was achieved; even after the software had been installed 
and approved, it took varying amounts of time until concrete benefits could be seen. This point was 
reached at different times in different agencies. 


The fact that the project was in the end limited to just four of the original cooperating countries 
somewhat simplified the task of project management, as much of the work involved agencies and 
even individuals who were old rather than new contacts. Nevertheless, beneficiary staff rotation 
was high (see Section “Progress against Outcomes”). 


Benchmarking and needs analysis 


The evaluation questions on benchmarking and needs analysis (see pp. 76 ff.) follow up on 
recommendations from the 2012 mid-term evaluation. 


Benchmarking is a structured process best known from the business arena in which workflows, 
approaches and performance are compared, numerically and qualitatively, between similar firms 
working in similar markets. It can be a useful way for firms to learn from one another. At the 6- 
monthly expert working group meetings hosted at CARICC, regional development was discussed, 
but a benchmarking approach as such was not applied. One factor which mitigated against the 
development of benchmarking was that project management was concerned to understand and 
adapt to the individual needs of each country. Furthermore, national agencies had their own 
standards, situations and regulatory frameworks. 


Nevertheless these are no longer completely persuasive arguments. Now that beneficiary agencies 
are more intrinsically motivated and persuaded of the benefits of the project model, there is perhaps 
a better environment for relaunching the idea of regional standards and a benchmarking exercise. 


Needs analysis: Alternative and additional software needs were discussed at the six-monthly 
experts meetings and suggestions were reviewed by CARICC, such as add-on packages and ways 
of bridging to other software. Project support was mainly centred around suggesting additional free 
software such as Google Earth. ARC GIS proved to be a useful complement to i2. Choice of 
software was also affected by the relatively limited funding available. The project was able to make 
some adaptations to the software solutions offered but did not have the technical capacity or funding 
to consider or indeed implement more fundamental changes to the approach. Since 2014 there was 
no dedicated IT expert within the project, but as the relevant expertise could be found within 
CARICC and the programme offices, this was not judged by beneficiary or other respondents to be 
an important drawback. 


Some agencies requested licenses for additional software e.g. for analysing social networks but 
these were turned down on the grounds of cost. Project management were further sometimes 
offered particular, paid-for software packages. Management however preferred to focus on well- 
established and free solutions. 


Summary of findings on Relevance 


The project was highly relevant to beneficiary needs and priorities. Improving the effectiveness of 
drug law enforcement is a priority and can be judged objectively to be an urgent and continuing 
need. The Outcomes were highly relevant to the project aims and the Outputs are an appropriate 
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way to achieve the Outcomes. In particular the role of intelligence-led policing (Outcome 3) 
became clearer to the relevant authorities through participation in the project. While beneficiary 
agencies do affirm the relevance of the project, it is unlikely that they would have implemented 
similar mechanisms without outside help. The long-term approach and partnership was an 
appropriate way to provide this kind of support in this region; the duration was unusually long 
considering that essentially the same project model was used throughout. The project was mainly 
concerned to adapt its approach to individual country and agency needs and benchmarking was 
not conducted systematically as part of ongoing regional development. A formal needs analysis in 
the strict sense was not carried out after 2012; however, informal, ongoing needs analysis was 
conducted through regular contacts with the beneficiary agencies and at the expert meetings. The 
ability of the project to purchase new software was seen as limited but agencies were encouraged 
to adopt some free software packages. 


Efficiency 


Evaluation questions: 

To what extent is the project implemented in the most efficient and cost-effective way compared 
to alternatives? 

Are the activities the most cost-effective way of achieving the outputs / outcomes / objective? 

Are there any good practices regarding efficiency, e.g. are certain aspects or arrangements of the 
project particularly efficient? 


Breakdown of costs; staffing history 


The total overall budget according to the last revision in 2014 was $8,016,445. The overall 
approved budget at the end of 2011 was $5,087,315, i.e. the overall budget for the evaluation period 
was $2,929,130 as per the costed workplan. Looking at costs in the costed workplan for the years 
2014-17, these were split approximately 50/50 between overheads — mainly staff costs — and direct 
project costs. The main items are shown in Figure 317. The two main headings are international 
staff and IT equipment. Approximately half of the costs were directly related to specific project 
activities, i.e. IT equipment, expert meetings etc. 


17 This is based on the budget breakdown 2014-17 as reported in last revision. The actual expenditure (according 
to generic headings and not the costed workplan headings) is shown in the Annex XI. 
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Figure 3. F23 Budget breakdown in costed workplan for 2014-17 
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The donors throughout the duration of the project were Austria, Italy, Canada, France, Norway, 
Turkey, United Kingdom and United States of America via their respective agencies, and UNDP. 
The main donor(s) for the phase 2012-17 was INL. 


It is important also to mention the national Ministries themselves as the main supporters of project 
implementation, as they did provide staff, administrative support, premises, running costs etc, 
without which it would not be possible to implement the F23 project. 


The staffing situation at the project end was as follows: The overall coordination of the project was 
in the hands of the Head of UNODC Programme Office in Tajikistan/International project 
Coordinator, supported by the Project Manager who was mainly concerned with project 
implementation. National staff was employed as part of the cost-share scheme: Project Manager - 
40%, Project Assistant - 100% plus a share of (security guards, IT expert, cleaner etc.) There were 
some small adjustments to national staff arrangements during the evaluation period, and from 2013 
to 2015 the project was managed by an Interim Project Coordinator/Project Manager due to lack of 
funding. 


Achievement of outputs 


Data on achievement of outputs was analysed on the basis of output-level information provided by 
project management — essentially, lists of people attending training. As the training courses were 
so central to the project, data for them is listed and analysed here. The evaluation team worked with 
the training records provided by project management, tagging the activities according to type, 
country etc. and then collating and analysing the records as displayed below. 
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Table 3. Numbers of F23 training courses held per year, by host country 


Grand 
Country 2012 2013 2014 2015 2016 2017 | Total 
Kazakhstan 2 1 1 1 2) 
Kyrgyzstan 1 1 4 2 8 
Tajikistan Ss 3 2 8 
(Outside 
evaluated 
countries) 4 4 
Grand Total 4 0 3 5 8 5 poe, 


A total of 25 training courses were carried out, approximately 4 per year, in all three countries 
except Uzbekistan. Most were held in Kyrgyzstan and Tajikistan, in response to agency requests. 
Additional training courses were provided in cooperation with other UNODC projects!8. 


Table 4. Numbers of participants* 


Grand 
2012 2013 2014 2015 2016 2017 Total 
Total 81 0 64 59 398 42 644 


*Note this total excludes over 700 additional participants from one single awareness-raising / information mission in 
Kyrgyzstan Northern Regions in 2017, as this was not a training course, and the involvement of these participants was 


more limited and not comparable to the other courses. 


Table 5. Number of training courses of different types conducted per year* 


Grand 
Data 2012 2013 2014 2015 2016 2017 Total 
Multi-country 2 0 2 1 2 0 7 
Multi-agency 2 0 3 4 7 3 19 
Includes different regions 
within same country 0 0 1 1 4 4 10 
Peer to peer 0 0 1 1 3 4 9 
On-the-job 0 0 0 1 1 0 2 
Train-the-trainer 0 0 0 1 1 0 2 


*Note that a “2” in the first cell means that 2 training courses were held in 2012 which were of a multi-country, etc. 
“Multi-country” means that beneficiary agencies from more than on country participated. One 2017 training in Kazakhstan 
with the participation of Kyrgyzstan and Tajikistan experts seconded to CARICC is not coded as “multi-country”. Note that 


one training course can appear in more than one row, e.g. a multi-agency train-the-trainer course. 


18 For example, an i2 Intelligence analysis training for LE officers/analysts from Tajikistan and Uzbekistan was 
conducted at CARICC in Almaty, June 3-8, 2013. The training was organized by the SP1 and K22 project 
with on-spot consultative and advisory support from the F23 project. This course is not included in the 
tables above. 
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The training courses covered a wide range of different needs. The number of on-the-job and train- 
the-trainer courses was not so high. On-the-job training was mostly provided if special support 
was required on a specific case and the agency permitted it (once confidential information was 
added to the database, subsequent support was usually provided remotely). 


Table 6. Trainings with participants from each country 


Grand 
Data 2012, 2013 2014 2015 2016 2017 | Total 
Kazakhstan 2, 0 2 2 2 1 9 
Kyrgyzstan 2 0 3) 2 5 2 14 
Tajikistan Z 0 2 3 5 2 14 
Uzbekistan 2 0 0 1 2 0 5 


Seven of the training courses included participants from more than one country (see previous table); 
over half the courses (14 out of 25) included participants from Tajikistan, and the same is true for 
Kyrgyzstan. However, only five courses included participants from Uzbekistan. 


Cost-effectiveness 


Respondents from partner governments and external respondents agreed that the project was 
managed efficiently and cost-effectively. Interactions between agency staff and project 
management were mostly smooth and timely. There were delays, however these were nearly always 
due to issues beyond the control of project management such as approval of equipment supplies. 


Procurement procedures can be very time-consuming, mainly due to centralised procurement and 
the new, UN-Secretariat-wide financial and administrative system Umoja. This centralised 
procedure is important for streamlining processes globally and to combat corruption but it does 
mean that procedures can be bureaucratic. Even once equipment was delivered, another layer of 
approval may be required before it can be used; this is a routine part of established procedure, with 
equipment remaining in boxes for several months, as in one case which the evaluation team took 
note of. 


Probably the biggest delays were due to all official communications between project management 
and the agencies needing to pass through the respective Ministries of Foreign Affairs. This is 
complicated by the fact that there are many related projects in the Central Asia region which the 
partner governments have to interact with. 


The project adapted its approach according to the needs of individual agencies, adding hardware 
and software licenses as and when appropriate. Hardware provision also included servers and 
networking where necessary. For example, the Analytical Unit of the Drug Control Agency in 
Tajikistan recently asked for a secure local area network to communicate with their Ministry of the 
Interior, which they received promptly. 
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International interchange of data (additional success dimension 4, see p. 3) was partly reliant on 
CARICC, which itself had to function within the challenging context of a multi-national, inter- 
agency centre. These challenges were no fault of F23. 


There were some examples of cost savings. National and international external partners reported 
that UNODC projects had worked together in an efficient way, and inspection of project records 
highlighted a number of partnerships!9. Training courses were further coordinated with other 


UNODC programme/projects to avoid overlap29. 


The version of 12 in use, mostly version 9, was relatively up-to-date compared to global practice in 
law enforcement agencies. Most of the agencies expressed satisfaction with the hardware; using 
computers which are not powerful enough for the software can itself be an inefficiency. 


The evaluation found that although the details of the project workplan were clear, a clearer overall 
project plan in the sense of the discussion in Annex VIII (inconsistencies between different 
statements of the project aims, no indicators or lack of indicators with SMART formulation, lack 
of baselines and monitoring data, etc) would have made it easier to gain a shared vision of exactly 
what the project was trying to do and what constitutes progress towards the vision. Although the 
agencies had access to the new Programme for Central Asia, it is regarded a large and complex 
Programme, where the different parts and their functions and inter-relationships were not clear to 
everyone. 


Summary of findings on Efficiency 


Achievement of outputs in most cases were in accordance with the project plans. Project activities 
were cost-effective ways of achieving outcomes, with the activities carried out during the evaluated 
timeframe building on initial steps taken from 2002 to 2012. There were some problems with delays 
but these were mostly not due to project management. 


Partnerships and cooperation 


Evaluation questions: 


What is the nature of the relationship between the UNODC project and beneficiary agencies and 
BCs? (e.g. could you describe it as ,,donor-driven“, ,, bottom-up“ etc)? 


Whose project is it? To what extent were stakeholders properly engaged and informed? 


19 F23 collaborated with the following UNODC projects: RER/H22 “Establishment of a Central Asian Regional 
Information and Coordination Centre (CARICC)”; XAC/K22 “Countering the trafficking of Afghan opiates 
via the northern route by enhancing the capacity of key border crossings points (BCPs) and through the 
establishment of Border Liaison Offices (BLOs)”; KGZ/K50 “Strengthening the State Service on Drug 
Control of the Kyrgyz Republic”; TAJ/H03 “Tajikistan Drug Control Agency (DCA) — Phase I”; KGZ/T90 
“Criminal Justice Programme in the Kyrgyz Republic” 

20 In September 2014, a training course on “Intelligence analysis training for the law enforcement analysts of 
DCA and MOI” was held together with KGR/K50 and TAJ/H03. In the same year, three other training 
courses were held in collaboration with CARICC, INL and/or the Coordination and Evaluation Unit of 
UNODC ROCA 
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How was the project conducive to the development of agency partnerships at the bilateral and 
multilateral level? This includes a) between agencies within same country and b) internationally 
and c) with other national and international partners. 


Was assistance continued at regional level in the current model, using joint activities, in house 
training and expert meetings? To what extent has UNODC F-23 sought / established 
partnerships with national and international partners, including agencies, UNODC-internally, 
public oversight bodies and civil society, etc. ? 


The agencies, according to most respondents, identified quite strongly with the project aims and 
approach but, with the some exceptions, regarded primarily as a vehicle for delivering technical 
support. So on the one hand it was mostly seen as mostly fulfilling genuine agency needs, and there 
was a good sense of partnership between the agencies and UNODC. On the other hand, there was 
less of a sense of ownership; F23 was seen as belonging to UNODC rather than to the agencies. 


On the issue of technical expertise, UNODC and the F23 project in particular were seen as major 
players with a technically sound portfolio. However the F23 project was seen as being relatively 
limited in resources. 


The project had to cover up to 18 agencies across several countries, which sometimes made it more 
difficult to adapt to individual needs and requests as well as the changing context (cyber-crime etc, 
dark net). The project followed correct procedures in engaging and informing partners, but a closer 
and more frequent contact would have been welcomed by most of the agencies; some compared 
F23 with some other UNODC projects which are supported by a National Officer in each country 
who assists with maintaining and facilitating in-country relationships and indicated that this 
arrangement would have been more desirable. 


Agency partnerships and regional assistance 


The project did not directly sponsor bilateral links; for example the individual agencies did not visit 
each other's offices in the framework of F23. The project did provide indirect support to bilateral 
linkages and multilateral cooperation; the most substantial example of a channel for international 
cooperation supported by F23 was CARICC. F23 strongly promoted and supported cooperation via 
this centre. Regional assistance was continued in all three areas via the regional meetings and in- 
house training, though there was only a small number of the latter. In-house or on-the-job training 
was mainly restricted to occasions when software was installed, which did not happen in the 
evaluated timeframe after 2012. 


Cooperation with other UNODC projects was mostly seamless, for example with TD/RER/H22 
"Establishment of CARICC" and XAC/K22 “Countering the trafficking of Afghan opiates via the 
northern route by enhancing the capacity of key border crossings points (BCPs) and through the 
establishment of Border Liaison Offices (BLOs)”. There was however little cooperation with public 
oversight bodies or civil society. 


Summary of findings on Partnerships 


This can be considered a relatively top-down project which however meets with the increasingly 
enthusiastic participation of the beneficiary agencies. Stakeholders were properly engaged and 
informed, but it was not always easy to maintain close contacts with so many agencies across four 
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countries. The project sponsored bilateral links indirectly but not directly. It was indirectly 
conducive to cooperation between relevant agencies in individual countries, and its support for 
CARICC contributed to very important regional networking. Regional level assistance was 
continued as planned. In-house training was restricted to specific needs on particular occasions 
though it was highly valued. 


Effectiveness 


Evaluation questions: 

To what degree did the project achieve its outcomes and objectives? 

What other major factors (apart from the project) helped or hindered? 

How good are beneficiary staff skills relevant to the 3 Outcomes? How much do the 3 outcomes 
actually happen in live operations (proportion of time?) 

To what extent was the project management (i.e. UNODC management & within stakeholders) 
effective and allowed implementing the set objectives under the project? 

To what extent did the project contribute to the Sustainable Development Goals? Was there any 
contribution to gender-specific SDGs in particular? 


Project effectiveness can be judged in terms of the essential and additional success dimensions as 
noted on p. 321 and which are derived from the project plans. Performance data from narrative 
reports and triangulated by interviews was compared against these dimensions in the following 
paragraphs. As argued on p. 3, to be judged as effective, within at least one participating agency 
within each country, with respect to trafficking of narcotics, the project would have to 
increase/enhance 1) information gathering and 2) analysis, and 3) this would have to be increasingly 
intelligence-led. Additional dimensions cover features which were suggested, but not formally 
required in the logical framework are: 1) achievement of the Outcomes in more than one agency in 
each country; 2) application: a) in particular to transnational narcotics trafficking and b) more 
generally to other organised crimes; and 3) increased exchange of intelligence between agencies 
within one country and increased exchange of intelligence between countries. 


The narrative within the mid-term evaluation report is a very useful description of the situation at 
project mid-term, for comparison with the current final situation encountered by this evaluation, in 
order to assess the contribution made by the project22. Overall it is very clear that substantial 


progress was made. 


Progress against Outcomes 


As noted already, review of the project document and revisions of it shows that the project was 
lacking clear definitions of indicator scores and in particular a systematic monitoring framework, 
or databases of such indicators (e.g. staff scores on a standardised skills test; prosecutions brought, 
etc) recorded at regular intervals over the life of the project. So there was no direct way to 
systematically assess key outcomes like staff skills, work done using 12 or other F23 outputs, or 
prosecutions brought as a consequence of F23 inputs, e.g. in which key evidence had been 


21 Since the indicators and targets given in the logical framework are not judged to be adequate without additional 
work, see p. 2 
22 Bearing in mind that some things might have changed even without the project 
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constructed with i2 — see recommendations on p. 37. No measures of staff skills — whether a test, 
or self-assessment supervisor or instructor assessment were applied, either as part of a regular 
assessment or even before and after the training courses. 


Ideally, the single most important set of data would be on how much the F23-trained skills and 
knowledge were actually used in participating agencies, were it available. Though there was some 
data on this in the project reports and some additional data was offered by agencies in interview, 
there was in fact no systematic monitoring database. The evaluation team discussed extensively 
with project management the possibility of systematically contacting the agencies to provide this 
data retrospectively for the evaluation, but this option was ultimately rejected as very unlikely to 
be successful in the evaluation timeframe. 


Outcomes 1 and 2: information gathering and analysis 


F23 contributed to information gathering and analysis skills within the agencies and these skills 
were put to use in nearly all the agencies visited at least once a week and often more frequently. 
This was only questionable for one agency visited and is not certain for the other agencies not 
covered by the evaluation (see Sample, p. 8). 


According to interviews and with reference to documentation provided, intelligence databases were 
in place and in regular use in nearly all the agencies. Link charts were created as a matter of course 
in many agencies. Telephone meta-data was the main content, but other types of data such as 
financial flows were also used. 


The project was able to make a difference on these key staff skills and behaviour in spite of the fact 
that there was a considerable amount of “rotation” amongst key staff, i.e. when staff moved to other 
positions. This problem is not unique to this region or this sector, F23 adapted to this problem and 
continued to accompany the agencies, attempting to provide advanced training for those who could 
benefit from it, and basic training for new staff. 


Several agencies said that on-the-job training was the most effective for them, although others also 
pointed out the importance of having a mix of training inputs, especially including basic training. 
As the F23 approach was able to offer a range of training inputs in negotiation with agencies, it was 
able to accommodate different needs of this nature, which was seen as very positive. 


It is remarkable that the majority of training courses were custom-designed to fit specific needs, for 
the individual needs of each agency, a country or the region, in order to maximise effectiveness. 


Selection of trainees was a key issue. Managers were tasked with identifying the right trainees for 
each course and were also involved in dialog with the F23 project management on their specific 
needs. However it was a recurring problem that agencies did not always nominate trainees who 
would most benefit from specific training courses or whose expertise could be put to the best use. 
It occasionally happened that attendance at a training course was approved as a kind of reward for 
specific staff — something that was seen by F23 project management as beyond their control. The 
absence of a specific F23 national officer in each country made it harder to monitor and optimise 
the process of nominating trainees. 


The F23 training approach reached a certain level of maturity. Nearly all agencies were familiar 


with the approach and the benefits it brought and assessed the courses as relevant, efficient and 
effective. Many respondents readily admitted that the time has come for individual countries and/or 
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agencies to take responsibility at least for basic i2 training, and some have begun to do that. (On 
the other hand, several agencies were still quite dependent on the F23-trained analysts.) There were 
many calls made by respondents for assistance to standardise at least the basic training approach, 
while underlining that each country and each agency has its own context, regulations, training 
academies etc. Respondents reported that there is some scope to integrate some of the content into 
existing training modules — for instance in law enforcement academies. 


Outcome 3 


Intelligence-driven operations and collaboration on live cases were lagging behind intelligence- 
gathering and analysis. The use of i2, and other F23 training inputs, certainly assisted in making 
agencies become more proactive. In comparison with the evaluation results of the mid-term 
evaluation, it is clear that progress was made. Interviewed staff were familiar with the concept of 
intelligence-led policing and could provide examples of where it was used, in particular in the sense 
of identifying possible new cases on the basis of existing data rather than reacting to crimes 
committed. There was less evidence that the project contributed to risk analysis or resource 
allocation being conducted on the basis of the tools provided. 


Summary of findings on progress against Outcomes 


With respect to the essential success dimensions identified on p. 3, the project certainly made at 
least some difference on all the three Outcomes in at least one agency in each country, with the 
biggest effects in relation to Outcomes | and 2. The situation on the “Additional dimensions of 
effectiveness” (see p. 3) was a little more mixed but overall very positive. 


- achievement of the Outcomes in more than one agency in each country 

As noted in the mid-term evaluation report, throughout the project, certain countries and Law 
Enforcement Agencies were more cooperative and more interested in adopting new practices than 
others; and the UNODC project staff made great efforts to include all beneficiary agencies at all 
times, individualising the approach in each case. 


It was very difficult to get a systematic assessment of the spread of engagement and progress across 
the beneficiary agencies, in the absence of systematic historical or current data relevant to this point. 
Discussions with project staff suggest that there was a good spread of involvement and progress at 
least across the four project countries (12 agencies), exceeding the implied minimum target of at 
least one agency per country; this was validated by the overall impression of the co-evaluator, who 
had some extensive knowledge of the project during part of the implementation period although 
she was not directly involved in it. However, it was not possible to corroborate this information 
further. 


- application a) in particular to transnational narcotics trafficking and b) more generally to 
other organised crimes 

Most of the narcotics cases relevant to F23 and dealt with by the agencies were in fact transnational, 
and representatives from two of the agencies reported that they were actively using the software for 
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non-narcotics cases23. So project influence was broader than the minimum area of operations 
implied in the project documentation. 


- increased exchange of intelligence between agencies within one country and between 
countries 

There are several well-known and understandable disincentives to sharing information. Probably 
the best single route for promotion for personnel is to solve an important case, so even individuals 
within the same agency may not want to share intelligence if they think they can finish it on their 
own. Similarly, one agency within a country may sometimes prefer to try to complete a case on its 
own than share intelligence with another agency. Also, agencies within one country obviously have 
different profiles; there is a certain amount of rivalry and one agency may not see that another has 
the right competences to add value to solving a case. Finally, sharing information is not always 
desirable or appropriate and may have negative consequences — in particular, it increases the chance 
of a leak, either because information channels are technically compromised or because personnel 
are not trustworthy. For example, an anticorruption agency might be well advised not to share its 
information as they might involve future targets within the same government. This fear is 
particularly strong with respect to exchanging intelligence between countries. 


The F23 project worked to facilitate information exchange on several levels. In Uzbekistan, F23 
had a particularly positive impact on cooperation between the agencies, linking also with the 
UNODC K22 project on border control. Secure communication was possible using the Intellect 
software provided by F23. Within Tajikistan and Kyrgyzstan, some of the agencies were part of a 
country-specific internal network with the support of F23, and Kazakhstan was also hoping to 
install such a network. 


Exchange of live intelligence between countries remains an area with considerable potential but 
also the most work still to do. CARICC is seen as the channel through which this exchange should 
take place. However, this is still a slow process. The main reasons for restricted performance are 
connected to national and international regulations and agreements; individual member states are 
held back by the challenge of differing national legislation; they take the risks of intelligence 
exchange very seriously. Member States do see the need for and benefits from regional cooperation. 


On average, CARICC facilitates intra-country information exchanges about once a week through 
the liaison officers24, as physical letters delivered in a diplomatic bag. Using formal channels, it 
can take up to two months for the requested data to return. CARICC has developed a proposal for 
an extension of the i2 software which would enable limited sharing of specific pieces of information 
between partners but it would cost several hundred thousand dollars and would still be subject to a 
multilateral information exchange framework agreement being signed by all parties. There is a 
multilateral agreement which has been drafted that still has not been signed, even though it is only 
a framework agreement which does not tie the partners to actually sharing any information. An 
alternative is to use the Interpol system i.e. de-classifying information before sharing. 


Apart from facilitating exchange of intelligence, CARICC also provides many useful strategic 
analyses although these are still not comprehensive, for example Uzbekistan is not properly 
included. 


23 This claim, though quite plausible, is difficult to verify. 
24 The liaison officers are situated in CARICC and the salaries are paid by UNODC 
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Gender 


The gender balance amongst analysts is still quite poor, and a few respondents expressed the 
opinion that law enforcement is a generally a more challenging environment for women mainly due 
to night work and travel requirements; however, these challenges apply less to the specific role of 
analyst which was therefore seen as being a role which it is easier for a woman to conduct in the 
four evaluated countries. 


Gender-specific records were not held for the training courses. However, a retrospective calculation 
was made by the evaluation team for the courses and expert meetings for 2012-2017, using the 
names of the trainees to determine gender, which suggests that about 10-11% of participants were 
women, with some improvement in 2017. 


Figure 4. Percentage of women on F23 training courses, by year 
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It was difficult for F23 to influence selection of participants - the invitations to attend the project 
events were sent through the MFAs to the agencies. They were verbally encouraged by F23 to 
involve more women, an encouragement which is recommended to also be made in writing. 


Awareness programmes and expert pairings 


Were comprehensive awareness programmes conducted amongst operational staff to ensure full 
understanding of intelligence-led law enforcement, its requirements and its benefits; - did they help? 


Was an international expert ‘team’/pairing established to mentor and provide skills transfer in situ 
with beneficiaries and on a rolling basis amongst them? 


These evaluation questions follow up a corresponding recommendation from the 2012 mid-term 
evaluation. 
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The answer to the question on awareness raising is yes, according to both internal records and 
interviews. Awareness-raising on modern intelligence work was seen as key not only for staff but 
also for management. It contributed to the mainstreaming of “the F23 approach” which occurred 
across many of the agencies. 


Both peer-to-peer and on-the-job training from experts from the region was held (see Table 5) in 
limited numbers and this was judged by agency respondents as having been very useful. 
Expert/team pairs were established in Tajikistan (DCA), Kyrgyzstan (MOI) and Uzbekistan 
(National Center), as well as at CARICC. 


Most F23 on-the-job (or “on spot’) trainings were provided at the initial time of software 
installation and creation of the data base, which was mostly in 2007 -2010. The on-the-job trainings 
were provided mostly if special support was required on a specific case and the agency permitted 
outside experts to view the data. As soon as the agencies started working with the data base and it 
contained confidential information, they mostly received support remotely. 


Project management 


The project management had to carry out a demanding task with relatively sparse resources (both 
the international Project Coordinator and the regional Project Manager also had roles in other 
projects). The project management was judged as above average in efficiency, meeting deadlines 
effectively. Most difficulties were attributed to the new UN-wide Umoja system, see p. 14. There 
was no project officer position in each country, and this was mentioned as a weakness by some 
respondents. Stronger project staff presence in the participating countries could have helped with 
speeding up exchange of information and could also have helped to identify specific training and 
support needs more rapidly, and could have assisted in selecting trainees. 


There is scope to further reduce the amount of travel using webinars and other online possibilities. 
However internet access is problematic in some agencies, and most respondents would not want to 
reduce the amount of personal contact much further, as face-to-face contacts were seen as being 
particularly important in the region. 


As previously mentioned, there was no systematic monitoring framework; the monitoring system, 
such as it was, was not really fit for purpose. The project annual reports did report to the indicators 
provided in the logframe. Baselines were provided except at Objective level. However as 
mentioned on pp. 3 ff., some of the indicators were problematic and many were not SMART. 
Although there were some detailed narrative reports, there was no monitoring framework or 
database to track the key indicators over time. Post-training questionnaires were used after training 
courses (and these show a very good level of satisfaction with the courses) but pre-tests were not 
used to compare pre-test and post-test performance. Results were recorded in individual files. There 
seems to have been little expert input to ensure logical consistency and quality of the logical 
framework and associated documentation; only the Strategic Planning and Inter-Agency Affairs 
Unit (SPIA) at HQ is involved in quality assurance. 
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Sustainable Development Goals 


Envisaged UNODC contributions to the SDGs were outlined in the relevant corporate document”. 
Narcotics interdiction should plausibly contribute to SDG 3.5 “strengthen the prevention and 
treatment of substance abuse”. (Further discussion of the effectiveness of an enforcement / 
interdiction approach as opposed to contrasting approaches like harm reduction is beyond the scope 
of this evaluation.) The project also plausibly26 contributed to SDG 10, as drug-related profiteering 


is a significant cause of inequality in Central Asia. More effective and more evidence-based 
prosecutions also means progress on SDG 16, ,,promote peaceful and inclusive societies ... Access 
to justice for all“, and specifically, TARGET 16.A — “Strengthen relevant national institutions, 
including through international cooperation, for building capacity at all levels, in particular in 
developing countries, to prevent violence and combat terrorism and crime”. F23 was part of 
UNODC support to Member States to reach the Targets under SDG 16 by improving criminal 
intelligence analysis and information management capacities, and further develop mechanisms for 
collection, evaluation, exchange and dissemination of information between drug control agencies 
in the Central Asian region. 


The findings in the Relevance and Effectiveness sections above provide support for the plausibility 
of these contributions to the SDGs. 


Summary of findings on Effectiveness 


It was a challenge for this evaluation to try to quantify progress on the three Outcomes. What can 
be ascertained is that the F23 training approach reached a certain level of maturity and was 
delivering good results in all three outcome areas, especially the first two. Analysts were more in a 
position to use the skills and capabilities they acquired through the project in their daily work. The 
approach propagated by F23 became a central part of law enforcement operations in some if not all 
agencies. “Rotation” of trained staff into other positions did happen to a considerable extent but 
with advantages for agencies as well as disadvantages. Summarising the progress on Outcomes, the 
project certainly reached the essential standards of progress implied in the logical framework (see 
p. 3) and also showed some additional successes. Turning to gender-specific effectiveness, the 
gender balance amongst analysts was quite poor. Gender-specific records were not held for the 
training courses. About 10% of participants were women, something which it was difficult for F23 
to influence; there was some improvement in 2017, when the percentage increased to about 16%. 
Following recommendations of the mid-term evaluation, awareness-raising was conducted for staff 
and also for management and was successful in helping to mainstream the F23 approach in many 
agencies. Expert mentoring was also carried out, but only to a limited extent; it was well received 
but relatively expensive. 

The project management succeeded in a demanding task with relatively sparse resources. However, 
the project monitoring system, such as it was, was not really fit for purpose. The logframe items, 
indicators, baselines and targets often did not match up with one another and many were not SMART. 


Finally, the project made plausible contributions to the SDGs and target 16.A in particular. There 
was no evidence of a substantial contribution to gender-specific SDGs. 


25 https://www.unodc.org/documents/SDGs/UNODC-SDG _ brochure _LORES.pdf 
26 There is no hard evidence for or against any of the points in this section. Complex research designs would be 
required to gather such evidence. The evaluation team only points out which connections are plausible. 
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Impact 


Evaluation questions: 

To what extent has the anticipated impact been reached by the project? 

Have there been any positive or negative unintended results? 

Any consequences on women and men differently? Any consequences for human rights? 


The project documentation did not use the word “impact” when setting out the project vision. 
However the “Expected Accomplishment” could be construed as impact: “1 (b) Member States are 
equipped to take effective action against transnational organized crime, including: drug trafficking; 
money-laundering; trafficking in persons; smuggling of migrants; illicit manufacturing and 
trafficking of firearms; and emerging policy issues as mentioned in General Assembly resolution 
64/179”. 


It is difficult to formally establish impact on drug trafficking per se. Possible indicators include 
number of seizures, street and wholesale prices of narcotics and so on. However all of these are 
problematic in their own way, quite apart from the fact that assessing the contribution of a small 
project like F23 to changes at this level is almost impossible. 


However it is possible to affirm that the project plausibly contributed to improved, evidence-based 
prosecutions: one national agency said they have disrupted the work of four transnational groups 
at least partially with the help of F23 and the i2 software, with 41 cases of groups of individuals 
brought to trial; another said that i2 had helped provide evidence in several hundred cases since 
2012. So the responsible officers in two relevant agencies in two of the four evaluated countries 
reported a substantial impact. In comparison with the narrative from the mid-term evaluation report, 
it seems likely that more narcotics prosecutions were being brought with the help of F23, using a 
better standard of evidence. 


There is also evidence for additional positive impact in several thematic areas close to the project 
which were not specifically anticipated. For example, F23 contributed to the Afghanistan 
Kyrgyzstan Tajikistan “Initiative AKT27”. 


F23 support to CARICC leveraged the reach of that organisation, such as a substantial number of 
relevant publications (more than a thousand info bulletins in the last ten years), 39 bilateral or 
multilateral operations in the last ten years, as well as CARICC staff conducting trainings in the 
region and beyond. 


The tendency towards more sustainable training models included moving some of the training 
expertise in-house through train-the-trainer initiatives. Teaching and training a subject was also a 
useful exercise for staff because it also helped them improve their own expertise. 


Before being included in a training course or other F23 initiative, candidates were subject to vetting 
via the US State Department. This was important for at least two reasons — first, analysts could 
become involved who were passively connected to criminal networks and for this reason did not 


27 “The Afghanistan-KyrgyzstanTajikistan (AKT) Initiative is aimed at strengthening cross-border cooperation in 
law enforcement and legal matters between these countries at the starting point of the Northern Route for 
trafficking narcotics from Afghanistan.” Source: 
https://www.unodc.org/documents/rpanc/RP_Success_Stories_distribution.pdf 
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pursue some targets, reducing project effectiveness. Second, analysts could be actively involved in 
criminal activity and could use insider knowledge of intelligence procedures to criminal ends. 


More fundamentally, the kind of approach and the hardware and software provided by F23 was in 
principle suited to controlling and suppressing not only criminal activity but also legal and illegal 
opposition groups — this technical possibility was confirmed in principle by interviewees; there is 
no evidence that this might be happening, but also no way to be sure that it is not. See p. 31. 


Support for CARICC ; integration of Customs Administrations 


These evaluation topics follow up on specific recommendations from the 2012 mid-term 
evaluation. 


Project documentation and interviews show that F23 had continued support of CARICC, 
specifically through activity 3.2.1, which envisaged regular expert meetings together with 
CARICC, working on a collective strategy and implementation for sustainable information 
exchange channels in the region, mostly via CARICC itself; and also more generally through its 
operational assistance for national law enforcement in the region. Interview with beneficiary 
agencies and CARICC staff confirm that F23 was seen as being a key partner by CARICC, with 
considerable influence but a fairly modest budget line. However, the longer-range effectiveness of 
this support depends on the effectiveness of CARICC itself which is beyond the scope of this 
evaluation. 


Respondents suggest that the sustainability of CARICC is likely to continue to be dependent on 
international support. There is a costed work plan for the activities of CARICC, and the F23 project 
was supposed to work towards its sustainability, but only Kazakhstan contributed a small amount 
as a voluntary contribution. A vision for CARICC, with a clear business model, outcomes and 
progress indicators, for both the coordination and intelligence-sharing functions, is a challenge 
which can be at least partly addressed by the new Programme for Central Asia. 


Greater interaction with customs services (and also with border services) was a logical step for F23. 
However, these services have very different needs and ways of working and already have their own 
databases which are less focused on criminal intelligence. Most seizures are not made at official 
border crossing points where customs agencies are situated, so interaction with state border services 
is important too. F23 recognised that a “one-size-fits-all” approach of simply promoting the i2 
software did not work. To at least some extent, F23 in collaboration with other agencies managed 
to promote a tailored approach which was most successful in Uzbekistan and to a lesser extent in 
Tajikistan. Through the K22 BLO project, F23 was able to work together with Boarder Liaison 
Officers who provided an interface between customs and State Border Services. 


Summary of findings on Impact 


The project plausibly contributed to a larger number of more evidence-based prosecutions, which 
was a significant achievement. There were some positive results in neighbouring thematic areas 
which were not directly intended. It is not possible to exclude some negative effects on human 
rights. F23 continued its valued support for CARICC, which continues to play an important role. 
There was some gradual progress in including Customs Administrations more formally but this 
still differs from country to country 
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Evaluation questions: 

What measures are in place to ensure future maintenance and repair of the facilities and 
equipment provided? 

To what extent are project interventions and impact sustainable in the longer term, say 3-10 
years? 

Were there continued efforts to secure high level and donor support by UNODC, partners and 
beneficiary agencies. 

How much donor visibility was there? 

Were efforts made by beneficiaries for self-funding? Was there direct outreach to potential 
donors? 


Computer hardware was a key part of the F23 approach, so its sustainability is crucial to the 
sustainability of project outcomes in relation to future similar projects in the region. An additional 
question arising from the mid-term evaluation (see Annex on p. 76) was whether necessary IT 
updates were fulfilled by use of replaced UNODC equipment and renewed donor funding. 


Stakeholders expressed puzzlement at the question on IT updates, because surplus UNODC 
hardware was not judged to be more advanced than agency hardware over the hardware lifecycle 
of the project. F23 provided hardware updates when necessary, involving all of the project countries 
in the last 12-24 months. Hardware updates were also provided from different sources including 
other UNODC projects. Interviews revealed that there was no specific systematic approach to future 
maintenance and repair on the part of the agencies, as each agency received tailored support in 
varying timeframes and hardware would become outdated at different times. This is probably the 
most cost-effective approach. It is difficult to be sure what would happen in the absence of further 
support from F23 or similar projects when hardware needs replacing. 


As far as sustainability of staff skills is concerned, many agencies said in interview that they follow 
a sustainability procedure according to which when staff leave a unit, their replacement is trained 
in equivalent skills (this means in particular, skills relevant to the F23 project). This training is 
conducted either by the person leaving or by other analysts; this would contribute significantly to 
sustainability of skills. 


Within all the project countries F23 helped improve institutional memory; people within the 
agencies left, but the long-term partnership with F23 helped reduce the risk that working practices 
introduced with the support of the F23 project might be forgotten. The project countries’ motivation 
to institutionalise elements of the F23 approach is, according to respondents, probably increasing 
as the benefits of the approach become more apparent. Individual staff were on the whole motivated 
to learn the skills which F23 brought and there were rewards for proficiency — not only promotion 
but in some cases medals and awards. Most respondents said that internal motivation was increasing 
throughout the life of the project. 


It should however not be forgotten that, as mentioned above, the national Ministries were really 
substantial supporters of project implementation, as they do already provide staff, administrative 
support, premises, running costs etc. 


Study visits, even to peer countries, can provide motivation by illustrating potential successes and 


the benefits of successful operations. The regional meetings can also be helpful in building this 
kind of shared vision as well as being a useful forum for keeping up with new developments. 
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The sustainability of the F23 approach really depends on national ministries being convinced that 
it benefits them. 


Sustainability of training model 


An additional question arising from the mid-term evaluation was whether staff recruitment, rotation 
and retention issues were addressed with beneficiary agencies. Many respondents spontaneously 
expressed the opinion that the training model was successful and was ripe to transition to a more 
sustainable model with much greater ownership nationally and regionally. External respondents 
agreed with the suggestion that there is room for more systematic work on training curricula and 
support on how to integrate them into existing national and agency-level training, even though the 
details would certainly look different from agency to agency. At least one agency already 
developed their own manual for the software with approval from the Ministry of the Interior and 
nominated their own trainers, with the aim to integrate the material within the IT program at the 
police academy. 


Thanks to F23 and the Train the Trainer component, there is now a significant pool of higher-level 
trainers who can form the core of training sustainability, perhaps moderated by CARICC. Trainers 
from the region are valuable, as they know situation on the ground. However there will still be a 
need for technical input, via similar inputs to those provided by F23, on more advanced issues, 
newer developments, etc.; to some extent CARICC can and does fulfil this function but the 
UNODC Programme for Central Asia could also help coordinate the provision of these inputs. 


F23 management did not get involved directly with HR issues such as recruitment and rotation at 
the agencies because their understanding was that there was very little that could be done, due to 
agency policies, and outside the remit of F23. The beneficiary agencies themselves were aware of 
these issues. Interviewees pointed out that the solution may not be to try to stop rotation, as it does 
have the benefit of spreading awareness and expertise throughout each agency, (staff trained in the 
i2 software and ways of working may go on to make net contributions to other projects). In fact it 
would be difficult to track individuals within organisations as they might see this as threatening 
and a problem of role confusion. However this does mean that it was more difficult for F23 
management to keep a track of key project outcomes. 


Funding 


The funding situation overall in the region is problematic, with an overall decrease in donor support 
for this type of intervention in the last few years in spite the continuing need. The range of potential 
donors is very limited. 


Kazakhstan is now rated as an upper middle income economy and an emerging donor”’, but the 
other project countries are all lower-income29 so are more dependent on continued support from 


international agencies. 
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There have been efforts on the part of UNODC to secure donor support in-country for continued 
support for the F23 model and also to engage donors for CARICC, which is extremely aware of the 
need for sustainability and the threat of gradually decreasing funding. The beneficiary agencies felt 
less responsibility for funding of the project itself, but were active in securing related and 
complementary inputs such as equipment from other sources in the context of other projects. They 
did approach UNODC with ideas and requests for sensible additions and extensions such as 
installing VPNs to interconnect their internal regions and agencies. 


There was little donor visibility. INL was sometimes visible at some meetings and events. However 
there was some considerable visibility for UNODC in the sense that all stakeholders (including 
CSOs) were aware of UNODC and F23. 


Future developments 


Government agencies expressed in interview that they are very concerned about new and emerging 
threats and issues, not only in terms of the linkages to narcotics trafficking but also as criminal 
developments in their own right, and elements of this concern also appear in National Strategic 
Plans. Key issues named by them and through literature review include: 


New encrypted communication applications e.g. WhatsApp. 
Dark Net 

New psychoactive substances 

Money laundering, offshore 

Online purchasing of illegal drugs 

Support for terrorism 

Cybercrime 

E-wallets, Bitcoin and similar currencies, Blockchain 


Some of these issues are already addressed by UNODC regionally and globally but agencies were 
specific in wanting to see them linked more closely to the F23 approach and the i2 software. 
Probably the most significant development is the first one. Some of the work of the agencies is 
based on telephone analysis but many criminal groups have moved to applications like WhatsApp 
with encrypted messaging. This transition is not complete yet — so ordinary telephone analysis is 
still very useful — but the trend is clear. This shift does not in any way invalidate the overall 
approach, as 12 software and ways of working applies just as much to other flows such as financial. 
The shift to encrypted communication increases the importance of physically seizing telephones 
from which communications can then be accessed. 


Heroin addiction amongst national populations is seen as a declining problem; existing and new 
users are moving to marijuana and other drugs as it is difficult to find opiates. Young people are 
reported to be increasingly interested in religion and sport as well as turning to other drugs, but on 
the other hand they may underestimate the risk of these other drugs. There is low but increasing 
use of synthetic drugs; a particular danger is the easy availability of Ephedra which grows wild 
across the whole region29, 


http://econ.worldbank.org/WBSITE/EXTERNAL/DATASTATISTICS/0,,contentMDK:20421402~menuPK: 
64133156~pagePK:64133 150~piPK:64133175~theSitePK:239419,00.html#Low_income 

30 ,, Trends in drug trafficking in Central Asia and Southern Caucasus, Nov 17, UNODC 
https://www.unodc.org/documents/regional/central-asia/Illicit/20Drug%20Trends_Central%20Asia- 
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Summary of findings on Sustainability 


It is highly likely that most agencies would continue with the use of i2 and intelligence-led policing 
within each country even in the absence of similar support; however, regional cooperation, mutual 
support in training etc, would be less likely to sustain. The training model was on the verge of 
becoming sustainable. The funding situation was problematic, even though UNODC made strong 
efforts to secure further funding. The agencies were able to secure indirect support from other 
donors, e.g. sharing equipment provided by other projects. UNODC, but not to the same extent the 
main donor (INL), had quite high visibility amongst the beneficiaries. Hardware updates were 
partially provided from F23 and other sources but not from replaced UNODC equipment as this 
was not judged a satisfactory solution. Recruitment, rotation and retention were taken seriously by 
the agencies, but were largely outside the remit of F23. Rotation was seen by the agencies not only 
as a threat but also an opportunity and a way of increasing project impact in new areas. All actors 
and especially government agencies were very aware of the rapid developments taking place and 
repeatedly asked for support with them. 


Human Rights & Gender Equality 


Evaluation questions: 

To what extent are human rights considerations included in the project development and 
implementation? 

To what extent are gender considerations included in the project development and 
implementation? 

Is gender-disaggregated data available? 

To what extent did the project intervention contribute to fulfilment of the National Action Plan on 
implementation of UN Security Council Resolution 1325 on women, peace and security? 

To what extent did the project intervention contribute to introduction and implementation of 
measures aimed at increasing women and minority representation in the police? 


Human Rights 


Human rights issues were not explicitly included in project design, training materials, or in the way 
the F23 approach was to be implemented; this is also evidenced by the fact that beneficiary agencies 
never mentioned human rights or gender issues in interview without prompting. Yet there are 
human rights issues which could well be addressed. For example, one agency said that there was 
no problem that their database is accredited with respect to personal data protection because it is 
“only used on suspects”. The possibility of threats to human rights is underlined reading the 
Kazakhstan National Development Plan, which explicitly discusses the possibility of reintroducing 
capital punishment for narcotics trafficking*'. So there are issues on which approaches like F23 
could be adapted to provide more input on human rights. 


final.pdf 
31 “Combatting narcomania and narcobusiness. We have to toughen penalties for import and spread of drugs, we should 
launch a discussion in the society - whether it is expedient to introduce capital punishment for that offence as is common 
in a number of countries including Malaysia and Singapore. Drugs are an altogether particular and destructive threat and 
it is quite a problem to what extent principles of humanism may be applicable here. In fact, on one scale we have the life 
of the man who imports and spreads drugs while on the other we see several lives ruined through the fault of that man.” 
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On the other hand, and more substantially, the F23 project was having a positive impact on this 
same human rights context, by increasing the extent to which prosecutions were based on evidence 
rather than, for example, confessions made under duress — see p. 26. 


The F23 project aimed to follow UN best practices on conflicts of interest and abuse of power. Staff 
undertook mandatory training to heighten their awareness. Procurement was done centrally through 
the Uzbekistan office, translators were taken from the UNDP roster and there were thresholds for 
suppliers. To mitigate actual corruption, all participants for training were vetted by the US 
government via INL. 


Gender Equality 


Gender equality issues were not explicitly included in project design, training materials, or in the 
way the F23 approach was implemented. The words “gender”, “female” and “women” were not 
mentioned anywhere in the annual reports or project revisions, except that the job profile for the 
International Project Coordinator attached to the latest revision mentions “Takes responsibility for 
incorporating gender perspectives and ensuring the equal participation of women and men in all 
areas of work.” As with human rights, respondents could not mention areas in which F23 had taken 
an approach which took gender into consideration. This is likely to be different with the new 
Programme for Central Asia, in which “Human rights and gender sensitivity will be embedded in 
all elements of the programme”, further in line with the general approach at UNODC with the 
mainstreaming of gender equality and human rights into all projects and programmes. 


But, as with human rights, the evaluation confirmed the need for a gender-sensitive approach. 
Female employment in law enforcement is traditionally low in Central Asia32. The number of 
women was reported to be increasing in many of the relevant agencies but mostly in lower-level 
and clerical jobs, though there are also some women in the analytical posts where barriers to 
employment may be lower. 


Gender disaggregated data was not collected as a matter of course in F23 documentation but could 
be reconstructed from names of attendees. The issue of the gender balance of trainees is further 
addressed on p. 23. There were no direct contributions to fulfilling Resolution 1325 or women and 
minority representation in the police. F23 did try to make sure that women are nominated for 
training courses. 


Summary of findings on Human Rights & Gender Equality 


The F23 project design did not specifically address gender or human rights considerations. 
However, the project most likely had a very positive impact on human rights by increasing the role, 
availability and value of evidence in the anti-narcotics process, supporting the rule of law. There 
might also be negative impacts in terms of enabling increased suppression of opposition groups 
but there is no evidence that this actually happens. There is some evidence that the human rights 
implications (data protection etc) needed to be made clearer in the training courses. Around 10% 
of trainees were women, a ratio which is typical for law enforcement in Central Asia. Gender 


Source: The Strategy «Kazakhstan 2030», pp 21-22, undated. Published on The General prosecutor's office of the 

Republic of Kazakhstan (http://prokuror.gov.kz). URL: http://prokuror.gov.kz/eng/printpdf/417 

32 http://www.ocamagazine.com/kyrgyzstan-seeks-gender-balance-in-law-enforcement; 
https://www.rferl.org/a/Do_Central_Asias_Gender_Quotas_Help_Or_Hurt_Women/1977535.html. 
Respondents also agreed that the ratio is, very roughly, 10%. 
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disaggregated data was not regularly collected. F23 made no direct contributions to fulfilling 
Resolution 1325 or to the percentage of women and minority representation in the police. 
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Il. CONCLUSIONS 


The project continues to be highly relevant to the changing situation on narcotics trafficking in 
Central Asia, from the perspective both of national governments and the international community. 
F23 fulfilled a need for outside expertise which on the whole the national governments could not 
have provided without support. The project made considerable effort to respect the individual 
situation and needs of each beneficiary agency. 


Looking back at the mid-term evaluation report, it is clear that the 2012-17 phase of the project was 
able to build on and expand the groundwork done in the first phase. It did not merely maintain a 
minimum level of capacity in each agency but on the contrary began to reap the benefits in terms 
of actual use of that capacity to facilitate evidence-based prosecutions. 


Achievement of outputs approximately followed the project plans, although already in 2012 the 
geographical scope of the project had to be restricted to just the four evaluated countries. Project 
activities were cost-effective ways of achieving outcomes. 


With respect to the minimum achievement criteria implied in the logical framework, see p.3, the 
project can be judged to have been extremely effective, and also added a lot of additional value 
along additional success dimensions such as including several agencies within each country and 
helping to improve interaction between them. To a lesser extent it managed to contribute to cross- 
border exchange of intelligence but there is still a lot of work to be done on that issue, some of 
which is beyond the remit of F23. 


Project support for CARICC contributed to very important regional networking. 


Training components have reached a certain level of maturity and national agencies are beginning 
to take over responsibility for them, with some material being included in basic material for Police 
Academy trainings. 


While some of the recommendations from the mid-term evaluation report were taken into account, 
more work could have been done to revisit them. In particular, in-house training, train-the-trainer 
and peer-to-peer training were efficient, effective and valued by partners but due mainly to funding 
restrictions were not used as much as suggested in the mid-term evaluation report. 


The project logical framework as formulated in the project document and revisions is perhaps its 
weakest point. The project management did have an implicit set of aims and a model of how the 
activities provided might lead to them, but it requires some work and a detailed reading of the 
project documents to really understand these. 


An explicit concern for gender and human rights considerations were missing from the project, and 
there were good reasons to include them even though they involve forces beyond the control of the 
project — firstly, the employment situation of women within the beneficiary agencies, and second, 
the human rights implications of collection and storage of personal data in the light of the 
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regulations in each country. On the other hand, the project probably had a very significant positive 
impact on human rights in terms of increasing the amount and quality of evidence in narcotics 
prosecutions. 
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IV. RECOMMENDATIONS 


These recommendations are largely interlinked but have been grouped under the following 
headings. As the F23 project has now ended, all recommendations are directed to the UNODC 
Coordinator for the Programme for Central Asia and _ staff responsible for the 
implementation of the successor activities to F23. 


Continue with approach implemented by the project 


The approach employed by F23 can overall be judged relevant, efficient and effective. The project 
certainly reached the essential standards of progress implied in the logical framework and also 
showed additional successes on a number of other areas. 


The key features (flexible provision of software, hardware and training, support for links between 
agencies, longer-term partnership with agencies, etc.) should be continued as part of the new 
Programme for Central Asia. 


Revisiting recommendations from mid-term evaluation in 2012 


Much progress was made with respect to the recommendations from the mid-term evaluation report 
— on expert/analyst training, trainer-training, in-house training, use of OSINT, databases and the 
holding of awareness sessions for key staff; subject to funding limitations. However they remain 
valid and most still need continued work. 


The following recommendations are therefore made: 


e Increase use of expert/analyst pairings, trainer-training, and in-house training, with sparing 
use of trainers from outside the region and increased use of trainers from inside the region. 


e Increase use of open-source intelligence (OSINT). 
e Continue to expand type of databases beyond telephone data. 


While awareness-raising sessions have been held within individual agencies as recommended in 
the mid-term evaluation, there could still be additional benefit from organising a special meeting 
for high-level staff (preferably deputy heads of Agencies), at which they could see the actual 
benefits of the F23 approach and could discuss the way forward not only within their countries, but 
also within the region. 


Sustainability of expertise; benchmarking 


There is a lack of concrete evidence of training effectiveness apart from satisfaction surveys, and 
no systematic way to formulate, assess or certify required skills and knowledge within or between 
agencies or countries. Given the continued need to anchor expertise within the region, and the fact 
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that agencies, with reservations, are interested in cooperation on regional standards and are more 
intrinsically motivated and persuaded of the benefits of the project model, there is perhaps a better 
environment for relaunching the idea of regional standards and a benchmarking exercise as 
recommended in the mid-term evaluation report. 


It is therefore recommended that a benchmarking exercise, possibly managed by CARICC, be 
conducted. This would start by identifying which F23-relevant analyst skills and equipment 
specifications could be specified as minimum requirements, and which need to be left to differ 
between agencies and between countries. Then training and equipment provision could be linked 
to these identified standards, with better target-setting, evaluation and scoring for training, and with 
the option to link to national training curricula and academies. Support could be provided to 
individual countries to implement their own training to these standards, and also mid-level and 
advanced areas of work could be defined where outside help is needed. Additionally, UNODC 
could suggest to the Member States that one leading agency within each country could be selected 
as a country-level "best practice" agency, with support to reach a specified minimum standard of 
excellence; UNODC could further support this process. More details on this recommendation are 
provided on p 99. 


Logical Framework and Monitoring 


The project logical framework as formulated in the project document and revisions is logically 
inconsistent and lacks SMART formulation. No systematic monitoring database of indicator scores 
(e.g. staff scores on a standardised skills test; prosecutions brought, etc) was kept or updated at 
regular intervals over the life of the project. 


It is therefore recommended that a concise and realistic logical framework is developed as the 
elements of F23 which interfaces with the minimum standards, above, and satisfies usual standards 
of evaluability and logical consistency, formulated ina SMART way and used for real-time project 
monitoring. Key outcomes need to be defined in a way which takes into account that the 
environment is changing even without the influence of the project. More details on this 
recommendation are provided on p 98. 


Emerging challenges 


There are many important emerging threats such as the use of encrypted messaging by criminal 
groups which seriously challenge the ability of agencies to respond to them, and the project’s 
procedures for dealing with these threats are not as fast as agencies desire. Yet the currently 
emerging challenges are so substantial that they cannot be considered merely optional extras. These 
kinds of developments are happening so quickly and changing the landscape that the project is 
currently struggling to support agencies fast enough. 


It is therefore recommended that project / programme management find a way to identify and 
address such issues in real time rather than over a period of years. Very frequent support from 
expertise outside the region will also be necessary for this, and there is scope for learning from 
“intermediate” countries such as Eastern and South-Eastern European countries. Key issues 
include: 


New encrypted communication applications e.g. WhatsApp. 
Dark Net 
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New psychoactive substances 
Money laundering, offshore 
Online purchasing of illegal drugs 
Support for terrorism 

Cybercrime 


E-wallets, Bitcoin and similar currencies, Blockchain 


Human rights 


Use of software and techniques provided and supported by F23 have some potential human rights 
implications (data protection etc) which are not dealt with thoroughly enough in the training 
courses; and the agencies are just beginning to come to terms with the ramifications of personal 
data protection, whether or not covered by corresponding national regulations and certification 
procedures. 


It is therefore recommended that training inputs are improved to strengthen their highlighting of 
human rights issues and that agencies are encouraged to work together on possible solutions; this 
should also involve components from other parts of the new Programme for Central Asia. 


Gender equality 


The project needs to do better on gender issues: the gender balance amongst analysts could be 
improved; gender is not specifically addressed in project design or training materials. Gender- 
specific records were not held for the training courses. 


It is therefore recommended that project/programme management should conduct an analysis 
of the employment environment in law enforcement — hindrances and enablers - related to gender, 
with special focus on the post of analyst, also considering disadvantaged regional groups within 
each country. Management should maintain gender-disaggregated data for training and expert 
meetings. Consider also modelling an inclusive approach in project implementation: identifying 
and working with key (female) individuals / champions who could act as change agents, at least 
one within each country and one at regional level. A similar approach could be used with human 
rights, identifying human rights change agents. 


Personnel 

Covering all the project countries and a growing number of beneficiary agencies is a challenge for 
centrally-based staff, and makes it difficult to maintain a close relationship with the agencies, for 
example to assist with and monitor selection of staff for training. 

It is therefore recommended, under the new Programme for Central Asia, to allocate a cost-share 


part of a National Officer post to ensure smooth and rapid relations within each country and with 
each agency. 


National integration 


There is a lot of scope for, and interest in, deepening integration between agencies within countries, 
including customs and border control, and some good progress was made in Tajikistan and also 


38 


LESSONS LEARNED AND BEST PRACTICES 


Uzbekistan. Deepening this integration could be relatively cost-effective as much expertise is 
already in place; support for encrypted data transfer may be an issue. 


It is therefore recommended that project/programme management continues to support deepening 


links between agencies in the same country, especially including customs and border control and 
including support to secure communication channels. 
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1. What are the lessons learnt from the completed “Drug law enforcement systems for criminal 
intelligence collection, analysis and exchange” project? 

2. What are the best practices that could be applied in the future activities and similar projects? 
3. What are the necessary adjustments to the UNODC interventions in the region within the 
undergoing UNOD Programme for Central Asian States 2015-2019? 


A notable best practice in the F23 project is was the exemplary long-term and individualised 
approach which accompanied national partners in a difficult context over more than one and a half 
decades. Patient and persistent help with applying the project model led to a major shift in the sense 
that agencies now see the benefits themselves and are motivated to continue the approach. 


Applying a similar approach on software and skills across agencies helped support vital coherence 
within agencies as well as within and between countries. This approach should be continued within 
the new Programme for Central Asia. 


The integration of the project approach into a broader program at this stage provides many potential 
benefits. For example joint steering committees, planned within the Programme, are not only about 
doing more with less, but should continue to deepen the existing links with other projects. 


Good integration between different projects within UNODC led to learning taking place between 
projects, e.g. on how to organise training courses. 


Recent simulated international cooperation exercises33 were a useful learning experience for 
national partners and highlighted legal problems which are now being addressed. 


Hands-on training inputs proved effective and were judged by trainees to be particularly relevant 
to their needs: they were held both “on-the-spot”, which proved useful to improve integration in 
actual workflows, especially in initial project phases, and also at venues remote from the actual 
workplace, which were valued because they shield trainees from day-to-day distractions. 


The support to expert meetings, alongside training and equipment, helped sponsor a core of key 
personnel with advanced level of expertise within the region, mainly but not only based at 
CARICC. 


33 Held during regional workshops on Promoting regional cooperation in Control Delivery and Joint Investigative 
Teams 
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ANNEX |. Terms of Reference of the Evaluation 


Background and Context 


Project title: Drug law enforcement systems for criminal intelligence collection, 
analysis and exchange 
November 2001 — June 2017 
Central Asian States, Afghanistan, Armenia, Azerbaijan and Georgia 


Linkages to Country, UNODC Programme for Central Asia 2015-2019: a partnership 

Regional and Framework for impact related action in Central Asia, Sub-programme 1: 

Thematic 1. Countering Transnational Organized Crime and Drug Trafficking 

Programmes: 1. (6) Member States are equipped to take effective action against 
transnational organized crime, including: drug trafficking; money 
laundering; trafficking in persons; smuggling of migrants; illicit 
manufacturing and trafficking of firearms; and emerging policy issues 
as mentioned in General Assembly resolution 64/179 


Executing Agency: UNODC ROCA 


Total Approved $6,333,630 
Budget: 


Total Overall Budget | $8,016,445 


Donors: Austria, Italy, Canada, France, Norway, Turkey, UNDP, United 
Kingdom and United States of America** 


Project Coordinator: Ms. Amelia Hannaford 


Type and time frame Final Independent Project Evaluation, 28 August 2017- 31 December 
of evaluation: 2017 

(Independent Project 

Evaluation/In-depth 

Evaluation/mid- 

term/final) 

Timeframe of the June 2012 — end of field mission (tentative: June 2017) 
project covered by the 

evaluation: 

Geographical Kazakhstan, Kyrgyzstan, Tajikistan and Uzbekistan 
coverage of the 

evaluation: 

Budget for this $50,000 

evaluation: 

Type and year of past | UNODC thematic evaluation of the project was conducted in 
evaluations: December 2006; 
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Core Learning Ministry of Interior of Kyrgyzstan Republic, Republic of Kazakhstan 

Partners? (entities): and Republic of Tajikistan, Drug Control Agency of Tajikistan 
(Analytical units/departments); CARICC Almaty, Kazakhstan, INL 
USA Embassies in Tajikistan and Kazakhstan 


Project overview and historical context 


The project’s objectives are to provide “a coherent and comprehensive system for intelligence, and 
information collection, analysis and exchange is established”. 


The project area of responsibility includes the five Central Asian countries Kazakhstan, 
Kyrgyzstan, Tajikistan, Turkmenistan and Uzbekistan as well as Afghanistan and Azerbaijan. 


Combating illicit drug trafficking requires well-organized systems of information/data collection, 
processing and analysis, as well as the exchange of the final information product among agencies 
involved at national and regional levels. 


Project F23 has improved criminal intelligence analysis and information management capacities, 
and further developed mechanisms for collection, evaluation, exchange and dissemination of 
information between drug control agencies in the Central Asian region, Afghanistan and 
Azerbaijan. However, the need to further develop a more sophisticated and proper mechanism for 
the collection, storage, exchange and analysis of information is required to fight against illicit drugs 
related problem in support to ongoing law enforcement investigations and has been set as one of 
the priorities for the continuation of F23 project’s activities fully supported by the attending 
countries. 


At present, counter narcotic agencies, cooperating within the frame of project F23, collect 
information and data in accordance with local requirements and needs. 


The F23 “project philosophy” is to teach similar working methodologies and using same analysis 
software and database structures for the law enforcement analysts within the participating countries. 
Such approach will stimulate national and regional cooperation between agencies and encourage 
the sharing of information and analytical products between Intelligence analysts from participating 
countries. 


The project has installed modern intelligence systems and i2 analysis software in 18 counter 
narcotic agencies in the project area. Approximately 400 officers have been trained in crime 
analysis techniques at various levels. All relevant agencies have analysts working with 12 software 
who are able to create analytical reports and visualization charts in support of criminal 
investigations and operational activities. 


It is necessary to continue developing a sustainable data and information collection capacity and 
root the Intelligence Led Policing implementation focusing on criminal intelligence analysis and 
information processing at counter narcotic Intel units. This exercise should also aim to identify 
priorities and a sound information gathering strategy for drug law enforcement bodies in each 
country. National drug agencies will be assisted in developing their information sharing and 
exchange system suitable to their objectives in combating illicit drug trafficking. 
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Main challenges during implementation 


The main government counterparts have shown commitment to the project and generally provide 
support towards implementation of project activities. The project team is able to have regular 
meetings and consultations with project beneficiaries to ensure proper implementation. 


Project implementation is monitored by a Coordination experts meetings consisting of 
representatives of project beneficiaries, other state bodies, donors and international organizations 
working in the field of intelligence analysis. 


The availability of accurate and timely statistical data on patterns and trends of crime is an 
essential prerequisite for the development of effective crime prevention policies. 
Afghanistan remains the main global opium producer with a crop yield of some 4,800 tons, 
compared to 3,300 tons in 2015 (+43%). Furthermore, Central Asia countries are used as 
major transit routes for drug trafficking on the so-called Northern Route. The situation in 
northern Afghanistan remains unpredictable, as there is a big concentration of Islamic State 
of Iraq and the Levant (ISIL, also commonly referred to as ISIS) very close to the Central 
Asian border, which might lead to an increase in the drugs flow from Afghanistan. 


More than ever before, the stakeholders in the Central Asia regions recognize the need and 
importance of expanding the efforts to address the complex challenges of drugs, crime and 
terrorism. Timely support to the law enforcement agencies (LEA) on countering narcotics 
and prevention of transnational organized crime is one of the key elements to peace, 
stability, and effective governance in the region, Moreover, it is important to stress that no 
country alone can address these complex problems and collaboration, joint programming, 
sharing of information, expertise and experiences are very crucial. 


Cybercrime, illicit financial flows and drugs online are seen as emerging concerns for the 
region; therefore, it is very important to give a special focus on increasing the analytical 
capacity of drug law enforcement and other relevant organizations in this region. Also, it 
is essential to increase the knowledge database and efficiency of decision makers on drugs 
related crime in the region. The central task of criminal intelligence analysis is to help law 
enforcement officers, policy makers and decision makers deal more effectively and provide 
timely strategic reports as “an early” warning of drug trafficking by organized criminal 
groups and to support operational activities by analysing crime. 


The need to continue developing a more sophisticated and proper mechanism for 
processing information and intelligence is obvious and has been set as one of the priorities 
for the F23 project’s activities fully supported by the Central Asian states. 


Project documents and revisions of the original project document 


Project document Year | General information regarding the original project document 
“Drug law | 2001- | Project F23 has improved criminal intelligence analysis and 
enforcement systems | 2017 | information management capacities, and further developed 
for criminal mechanisms for collection, evaluation, exchange and 
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intelligence 
collection, analysis 
and exchange” 


dissemination of information between drug control agencies of 
the project member states. 


Project revision 


Year 


Reason & purpose Change in 


Revision 1 


2003 


In line with the findings of the assessment | Budget 
mission (October 2002) the project was | Timeframe 
revised in terms of scope, duration and | Logframe 
budget. The revised project envisages 
provision of more advanced training, 
organization of study tours and seminars on 
intelligence-led policing, procurement of 
software packages, as well as assistance in 
drafting interagency agreements on 
intelligence collection, analysis and 
exchange. The revised overall budget 
increased to USD 1,998,600 


Revision 2 


2005 


UNODC and UNDP signed an agreement | Budget 
(POLINT) that coordinates the activities of | Timeframe 
this UNODC project with new and related | Logframe 
action executed by UNDP under the EC- 
funded CADAP programme. Thus, the 
current project revision addressed the 
provisions of this 2004 UNDP-UNODC 
MoU, expanded project duration, and 
refocused remaining activities of RER/F23 
on those components not covered under 
CADAP. 

This revision did not change the conceptual 
approach, legal context and implementation 
arrangements of the approved project 
TD/RER/F23, unless specified in the 
document. 


Revision 3 


2007 


Increase the initial life span and shape the | Budget 
Timeframe 


project activities to address new 
Logframe 


developments in Central Asia and 
neighbouring regions. The revision added 
two new outputs that were extending the 
country scope and fortify past results of the 
project. The first of the new outputs 
incorporated Afghanistan and Azerbaijan 
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into the F23 project. The second enhanced 
the project’s positive results by 
implementing additional intelligence-led 
policing components in countries that were 
involved in the previous phases of the 
project. This revision did not change the 
conceptual approach, legal context and 
implementation arrangements of the 
approved project TD/RER/F23, unless 
specified in this document. The revised 
overall budget increased to USD 4,212,300 


Revision 4 


2009 


Given successful delivery of the project 
assistance and in line with the interest of the 
governments of Azerbaijan and 
Afghanistan to participate in the project 
activities, the project was revised in 2007 to 
extend geographic scope of the project and 
increase its initial life span. 


It was expected that the project would 
continue activities once the project revision 
has been signed by recipient countries. 
Unfortunately, the national clearance 
procedures have taken longer time then was 
initially expected so that some project 
activities commenced with some delay. 


In addition, in view of the increased scope 
of responsibilities it was suggested to 
upgrade project coordinator post to a more 
senior level, but this activity has not been 
materialized. 


Budget 
Timeframe 


Logframe 


Revision 5 


2010 


The fifth revision of the project has 
installed modern intelligence systems and 
analysis software providing operational 
analysis training at counter narcotic 
agencies in all five Central Asian countries. 
However existing mechanisms for 
collection, storage, exchange and analysis 
of information, including access to 
information sources, remain in need of 


Budget 
Timeframe 


Logframe 
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further support in order to achieve their 
potential. 


The Central Asian Regional Information 
and Coordination Centre (CARICC) are 
still in the process of developing its 
functionality after its inauguration and F23 
continues to play an important supporting 
role in that development through its 
technical and operational assistance for 
national analytical capacity. The collection 
and dissemination of criminal intelligence 
at the national level is a prerequisite for 
successful regional intelligence 
cooperation. 


The challenges associated with supporting 
the rule of law in Afghanistan are self- 
evident. This project is contributing to 
international efforts by helping the Counter 
Narcotic Police Afghanistan (CNPA) to 
develop a criminal intelligence capacity. 
Existing activities are on-going and it is 
proposed to extend them to include the 
Afghan Criminal Justice Task Force 
(CJTF). Because of the security situation, 
such activities are inevitably protracted and 
progressed cautiously. 

The project objective and output remain 
unchanged. The revised overall budget 
increased to USD 5,087,316 


Revision 6 


2012 


This project revision intended to increase 
the initial life span and shape project 
activities to address new developments in 
the regions. The revision added one new 
outcome that will extend the country scope 
and fortify past results of the project. The 
new outcome incorporated Armenia and 
Georgia into the F23 project. The revision 
allowed the project to continue to 
contribute to establishing a solid base for 
law enforcement agencies in the region to 
become self-sufficient in processing and 


Budget 
Timeframe 


Logframe 
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sharing information/intelligence, and reach 
a capacity where project trained analysts 
will be able to provide intelligence analysis 
training. It did not change the conceptual 
approach and implementation 
arrangements of the approved project 
TD/RER/F23. The revised overall budget 
increased to USD 6,263,686 


Revision 7 2014 | This project revision intends to increase the | Budget 
initial life span and shape the project | Timeframe 
activities to address new developments in | Logframe 
the regions. The revision allowed the 
project to continue contributing to the 
establishment of a solid base for law 
enforcement agencies in the region to 
become self-sufficient in processing and 
sharing information/intelligence, and reach 
a capacity where project trained analysts 
will be able to provide intelligence analysis 
training. The revised overall budget 
increased to USD 8,016,445 


Main objectives and outcomes 


The project is aligned with key national development priorities of the project member 
states, as set out in the following policy documents and laws. 


The objective of RERF23 is defined as: A coherent and comprehensive system for 
intelligence, information and information collection, analysis and exchange is established. 
It is supported by three outcomes: 


Specialized law enforcement personnel use enhanced management of information 
gathering and exchange systems. 
Law enforcement personnel use enhanced analytical capacity in operational work. 
Law enforcement personnel increase use of intelligence-led policing tools in 
operational work. 


The outputs include delivery of capacity building activities, infrastructure development, 
equipment provision, and cooperation mechanisms to enhance the drug law enforcement 
systems for criminal intelligence collection, analysis and exchange. 


The Project implementation has been regularly monitored against the set baselines, targets 
and indicators via internal project monitoring and reporting mechanisms: statistical 
disaggregated data of the project implementing partners, work plans, guidelines and policy 
documents’ expert analysis, coordination and steering committees’ meetings, site visits to 
project locations, project and monitoring reports. 
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Contribution to UNODC’s country, regional or thematic programme 


Programme for Central Asia, a partnership framework for impact related action in 
Central Asia 2015 -2019 


Sub-programme 1: Countering transnational organised crime, illicit drug trafficking and 
preventing terrorism. 


Regional Programme for Afghanistan and Neighbouring Countries 2011 -2019 


Outcome 1: Enhanced regional cooperation and coordination to address transnational drug 
- related crimes. 


The Project also contributes and is linked to the Sustainable Development Goals (SDGs) 
especially Goal 16, target 16. A -strengthen relevant national institutions, including 
through international co-operation, for building capacity at all levels, in particular in 
developing countries, to prevent violence and combat terrorism and crime. The countries 
are in varying stages of developing their national plans for the SDGs and UNODC 
contributes as part of the UN country teams. 


Linkage to UNODC strategy context and Sustainable Development Goals 


Within the United Nations’ and UNODC’s global strategic framework and based on the 
culture of shared responsibilities, collective action and benchmarking for progress the UN 
Sustainable Development goals, UNODC ROCA’s contribution provides support to 
Member States to reach their Targets specifically under the following Sustainable 
Development Goal: 


TARGET 16.A - Strengthen relevant national institutions, including through 
international cooperation, for building capacity at all levels, in particular in 
developing countries, to prevent violence and combat terrorism and crime: UNODC 
will support Member States to reach the Targets under SDG 16 by improving criminal 
intelligence analysis and information management capacities, and further develop 
mechanisms for collection, evaluation, exchange and dissemination of information 
between drug control agencies in the Central Asian region, Afghanistan and Azerbaijan. 
However the need to further developing a more sophisticated and proper mechanism for 
the collection, storage, exchange and analysis of information and data results to fight against 
illicit drug trafficking related problem in support to ongoing law enforcement investigations, has 
been set as one of the priorities for the continuation of F23 project’s activities fully 
supported by the attending countries. The F23 project is going to be integrated into the 
Programme for Central Asia, a partnership framework for impact related action in 
Central Asia 2015 -2019, Sub-programme 1: Countering transnational organised 
crime, illicit drug trafficking and preventing terrorism, as the UNODC Regional 
Programme for Central Asian States 2015-2019 signed in May 2015 by the CA 
(Kazakhstan, Kyrgyzstan, Tajikistan, Turkmenistan, Uzbekistan) government 
representatives and based on the integrated programming approach which aims to deliver 
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outcomes and outputs through sub programmes rather than through standalone projects and 
initiatives. 


This project is part of the UNODC Strategic Framework that works with Member States to 
enhance their responses to the interconnected problems of drug use, illicit drug trafficking, 
trafficking in human beings and firearms and, transnational crime, corruption and 
terrorism. To achieve this, UNODC helps Member States to create and strengthen 
legislative, judicial and health systems to better safeguard some of the most vulnerable 
persons in society. 


The project is implemented under the Sub-programme 1 UNODC Strategic Framework for 
2016 — 2017, countering illicit drug trafficking and transnational organized crime, to 
promote and support effective responses to transnational organized crime, illicit trafficking 
and illicit drug trafficking by facilitating the implementation at the normative and 
operational levels of the relevant United Nations conventions 


The Project contributes to this initiative by improving criminal intelligence analysis and 


information management capacities, and further developed mechanisms for collection, evaluation, 
exchange and dissemination of information between drug control agencies in the region. 


Disbursement History 


Time periods throughout 


rere ira oe oo Expenditure Expenditure in % 
2017) 

Overall Budget Total Approved Expenditure Expenditure in % 

(as of 30.06.2017) Budget (as of 28.02.2017) | (as of 28.02.2017) 
USD 8,016,445 USD 6,333,630 USD 5,594,945 94 % 


Time period of the project 
covered by the evaluation 


Allocation for the 
time period 


Expenditure 


Expenditure in % 


January 2012 — June 2017 


USD 1,507,000 


USD 1,333,104 
(as of 28.02.2017) 


88.46 % 


49 


Purpose of the Evaluation 


Reasons behind the evaluation taking place 


Pursuant to UNODC evaluation norms and standards, as well as the project donors’ 
requirements, apart from the mid-term evaluation already undertaken, a final Independent 
Project Evaluation is mandatory and is to take place prior to the financial closure of the 
project. 


The evaluation timeframe will cover project activities conducted over the period from 
January 2012 (since the end of data collection for the mid-term evaluation) to the end of 
the evaluation field mission (tentatively end of June 2017). 


The purpose of this final evaluation is to assess the evaluation criteria of relevance, 
efficiency, partnerships and cooperation, human rights and gender mainstreaming, with a 
particular focus on effectiveness, impact and sustainability, as well as assess the 
implementation of the recommendations from the mid-term Independent Project 
Evaluations in 2012 and derive lessons learned, best practices and recommendations for 
future project interventions to ensure ownership, result-based orientation, cost- 
effectiveness and quality of the UNODC services. 


The results of this summative final evaluation are intended for use by the UNODC Regional 
Office for Central Asia, Project Team, beneficiary agencies and Donor Countries. In 
particular, it will serve as a reference source for the lessons learned from the UNODC 
“Drug law enforcement systems for criminal intelligence collection, analysis and 
exchange” project, for its proper completion and integration with the necessary adjustments 
to the UNODC interventions in the Central Asian region within the undergoing UNODC 
Regional Programme for Central Asian States 2015-2019 signed in May 2015 by the CA 
(Kazakhstan, Kyrgyzstan, Tajikistan, Turkmenistan, Uzbekistan) government 
representatives and based on the integrated programming approach which aims to deliver 
outcomes and outputs through sub programmes rather than through standalone projects and 
initiatives. 


This final evaluation will be carried out by two Independent external Evaluators, with 
logistical arrangements provided by the UNODC Programme Office in the Republic of 
Tajikistan and in line with the UNODC evaluation policy, norms, standards, guidelines and 
templates. 


Assumed accomplishment of the evaluation 


Through this evaluation, UNODC ROCA will obtain an independent and objective 
assessment on the effectiveness of the activities conducted in this specific area and draw 
on recommendations and lessons learned to inform future programming. . Furthermore, 
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this evaluation will assess the progress made in implementing the recommendations from 
the previous mid-term evaluation in 2012. 


Specific proposed questions, among others include ‘To what extent have the resources 
available been converted to outputs in a timely and cost-effective manner for the 
knowledge products?’; “To what extent has the Project improved comprehensive system 
for intelligence, information and information collection, analysis and exchange?’; and ‘To 
what extend are the project results (outcomes and impact, if any) likely to continue / be 
sustained after the project has finished?’. 


This evaluation will provide for an opportunity to learn lessons for future UNODC 
interventions in the region; to provide accountability to donors by assessing the extent to 
which project objectives were met and resources were wisely utilized; and to suggest areas 
of improvement. 


The main evaluation users 


The main users and benefactors of this evaluation will be the UNODC Regional Office for 
Central Asia and Project management, Core Learning Partners (see Annex 3) and the 
project donors and beneficiary agencies of the project member states. 


The main stakeholders (CLPs) will be specifically selected, representing a balance of men, 
women and other marginalised groups (to be further refined by the evaluation team) in 
order to review and provide comments on the Terms of Reference and the evaluation 
questions,; review and provide comments on the draft evaluation report, as well as facilitate 
the dissemination and application of the results and other follow-up action. Their 
comments, opinions and ideas shall be reflected in the report where deemed appropriate by 
the evaluation team. The list of CLPs is found in Annex 3. Stakeholders include all those 
to be invited to participate in the interviews and surveys, including the CLPs. 


Scope of the Evaluation 


Unit of analysis 

(full project/ 
programme/ parts of the 
project/ programme; 
etc.) 


This final evaluation covers the activities of the RER/F23 project 
implemented in Kazakhstan, Kyrgyzstan, Tajikistan and Uzbekistan. The 
purpose of this final evaluation is to assess the evaluation criteria of 
relevance, efficiency, partnerships and cooperation, human rights and 
gender mainstreaming, with a particular focus on effectiveness, impact 
and sustainability. 


Time period covered 
by the evaluation 


Activities conducted over the period from January 2012 until the end of 
the evaluation field mission (tentatively end of June 2017). 


The scope for the geographical coverage of the project will be 
Kazakhstan, Kyrgyzstan, Tajikistan and Uzbekistan, as_ project 


Geographical priority/pilot locations. Missions to Kazakhstan, Kyrgyzstan, Tajikistan 
coverage of the and Uzbekistan are proposed, consisting of meetings with national 
evaluation counterparts in Astana, Almaty, Bishkek, Dushanbe and Tashkent. Exact 


details of field mission, however, to be further refined and discussed with 


the evaluation team. 


Key Evaluation Questions 


Evaluation Criteria 


The evaluation will be conducted based on the following DAC criteria: relevance, efficiency, 
effectiveness, impact and sustainability, as well as partnerships and cooperation, gender and human 
rights and lessons learned. The questions will be further refined by the Evaluation Team. 


Relevance 
Relevance is the extent to which the aid activity is suited to the priorities and 
policies of the target group, recipient and donor. 


1. To what extent is the project relevant to the respective beneficiaries’ agencies needs and 
priorities? 


2. To what extent do the objectives, outcomes and outputs respond to present circumstances and 
stakeholder expectations? 


3. To what extent have recommendations on relevance from the previous mid-term evaluation in 
2012 been implemented? 


Efficiency 
Efficiency measures the outputs - qualitative and quantitative - in relation to the inputs. 


1. To what extent is the project implemented in the most efficient and cost-effective way 
compared to alternatives? 


2. Are there any good practices regarding efficiency, e.g. are certain aspects or arrangements of 
the project particularly efficient? 


3. To what extent have recommendations on efficiency from the previous mid-term evaluation 
in 2012 been implemented? 


Effectiveness 
Effectiveness is a measure of the extent to which an aid activity attains its objectives. 


1. To what degree were the programme’s outcomes and objectives achieved? 


2. To what extent was the project management structure effective and allowed implementing 
the set objectives under the project? 


3. To what extent have recommendations on effectiveness from the previous mid-term 
evaluation in 2012 been implemented? 


4. To what extent did the project/programme contribute to the Sustainable Development Goals? 


Impact 
Impact is the positive and negative changes produced by a development intervention, 
directly or indirectly, intended or unintended. 


1. To what extend has the anticipated impact been reached by the project? 


2. Have there been any positive or negative unintended results? 


3. To what extent have recommendations on impact from the previous mid-term evaluation in 
2012 been implemented? 


Sustainability 
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Sustainability is concerned with measuring whether the benefits of an activity are likely to 
continue after donor funding has been withdrawn. 


1. What measures are in place to ensure future maintenance and repair of the facilities and 
equipment provided? 


2. To what extent are project interventions sustainable in the long term? 


3. To what extent have recommendations on sustainability from the previous mid-term 
evaluation in 2012 been implemented? 


Partnerships and cooperation 
The evaluation assesses the partnerships and cooperation established during the project/ 
programme 
as well as their functioning and value. 


1. To what extent were stakeholders properly engaged and informed? 


2. How was the project conducive to the development of partnerships at the bilateral and 
multilateral level? 


3. To what extent have partnerships been sought with national and international partners, 
including UN-agencies, UNODC-internally, public oversight bodies and civil society, etc.? 


4. To what extent have recommendations on partnerships and cooperation from the previous 
mid-term evaluation in 2012 been implemented? 


Human rights 
The evaluation needs to assess the mainstreaming of human rights aspects 
throughout the project/ programme. 


1. To what extent are human rights considerations included in the project development and 
implementation? 


2. To what extent have recommendations on human rights from the previous mid-term 
evaluation in 2012 been implemented? 


Gender 
The evaluation needs to assess the mainstreaming of gender aspects 
throughout the project/ programme. 


1. To what extent are gender considerations included in the project development and 
implementation? 


( 


2. To what extent did the project intervention contribute to fulfilment of the National Action 
Plan on implementation of UN Security Council Resolution 1325 on women, peace and 
security? 


To what extent did the project intervention contribute to introduction and implementation of 
measures aimed at increasing women and minority representation in the police? 


3. To what extent have recommendations on gender from the previous mid-term evaluation in 
2012 been implemented? 


Lessons learned and best practice 
Lessons learned concern the learning experiences and insights that were gained throughout the 
project/ programme. 


1. What are the lessons learnt from the completed “Drug law enforcement systems for criminal 
intelligence collection, analysis and exchange” project? 


2. What are the best practices that could be applied in the future activities and similar projects? 


3. What are the necessary adjustments to the UNODC interventions in the region within the 
undergoing UNODC Regional Programme for Central Asian States 2015-2019? 


Past Evaluations 


53 


The origins of the F23 project and its design lie in the UN Office of Drug Control & Crime 
Prevention (ODCCP) strategy of the time, which was itself governed by the Political Declaration 
adopted at the United Nations General Assembly (UNGASS) 20th Special Session which was held 
in June 1998. The UNGASS resolution Measures to Enhance International Cooperation to Counter 
the World Drug Problem encouraged “/.../ states in cooperation with competent international and 
regional bodies” to address the issues of data collection and information exchange. 


At a meeting held in September 1999, at the initiative of the Uzbekistan government, the Six plus 
Two Group (the six countries bordering Afghanistan, namely China, Iran, Pakistan, Uzbekistan, 
Tajikistan and Turkmenistan along with the USA and Russia) expressed their concern at the 
increased cultivation, production and trafficking of illicit drugs in and from Afghanistan. In 
September 2000 at a high-level meeting of this group, a regional action plan on countering the 
Afghan drug trade was drawn up in consultation with ODCCP. This regional action plan approved 
by all the members encompassed a set of objectives to combat the Afghan drug trade in the fields 
of law enforcement, regional cooperation, criminal justice systems, eradication of illicit drug crops, 
alternative development, counter-narcotics efforts within Afghanistan, demand reduction and 
demand prevention. 

From this it can be seen that a clear concern had already been identified globally and certain, 
concrete, high-level steps taken with the ODCCP involved from an early stage. 

At around the same time, The Programme Framework for International Cooperation on Drug 
Control in the Economic Cooperation Organization (ECO) Countries (Afghanistan, Azerbaijan, 
Iran, Kazakhstan, Kyrgyzstan, Pakistan, Tajikistan, Turkmenistan, Turkey and Uzbekistan) was 
drawn up and this provided the framework for the F23 project which ODCCP was now preparing. 
The programme was divided into 2 strands: 

. Strengthening (to the extent possible) the drug control capacities of individual countries 

. Addressing the drug problem in the ECO region through jointly coordinated action by 
countries using as far as possible the regional cooperation mechanisms. 

At the 5th ECO summit held in May 1998 the majority of the heads of state/government recognized 
the importance of concerted action within the ECO framework to pre-empt the rapidly escalating 
regional drug problem. 

The Programme Framework, which was presented to major donors in October 1998 and aimed at 
providing response to the increasing appeals from all ECO countries for drug control assistance. 
The Programme Framework also aimed at addressing the drug problem in the world’s largest 
opium producing, consuming and trafficking region through an effective use of the cooperation 
mechanisms. 

The Declaration and Priorities for co-operation to counter drugs, organised crime & terrorism in 
Central Asia adopted during the International Conference on Enhancing Security & Stability in 
Central Asia: An Integrated Approach to Counter Drugs, Organized Crime & Terrorism held in 
October 2000 indicated a genuine willingness of the Central Asian countries to improve 
cooperation and coordination, a readiness to develop systems for the collection, analysis, sharing 
and evaluation of data on drug trafficking and drug abuse and the establishment of mechanisms for 
gathering, sharing and exchange of information between all relevant agencies and bodies. 

As can be seen in the original project document, a number of key areas were identified as being 
necessary for the successful implementation of the project. In summary they covered strategy 
development, adoption of standardised intelligence methodologies, establishment of dedicated 
operational intelligence analysis capacity, the supply of equipment, training, mentoring and the 
development of means for cooperation and information exchange. Three clear and logical outputs 
were formalised and incorporated into the project document; 

Output 1 - A coherent and comprehensive system for data and information collection, analysis and 
exchange is established. 

Output 2 — Specialist law enforcement personnel are properly trained in the management of 
information gathering and exchange systems. 
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Output 3 — A mechanism for information gathering and exchange is established at a regional level. 
For each output, a significant number of specific and detailed activities were formulated, breaking 
down the outputs into manageable elements, taking into account the scope of the project, the 
difficulties of implementing regional projects and the low starting point of many of the BAs. 
Indeed, the overall philosophy developed of parallel development of individual BAs under a 
regional framework was an eminently suitable one and has proved effective. 

The project design identifies the pressing need to have full support and, crucially, the understanding 
of BC governments and BA senior management in the adoption of intelligence-led law enforcement 
and policing methods as operated in developed agencies and services around the world. The project 
proposed achieving this through the sensible step of a series of comprehensive and targeted 
workshops to be conducted at the commencement of the project. The workshops would also be the 
support tool for the exchange of views, agreement of requirements, identification of problems and 
beginnings of the establishment of national and international networks. This step at the outset of 
the project was vital to all the work that followed. 

The project design also took into account the lessons learned from past experience in the delivery 
of assistance in the area of intelligence. The lessons identified included the needs for systematic 
delivery of assistance and genuine coordination between those delivering assistance. Projects such 
as the EU-funded Central Asia Drugs Action Plan (CADAP) and National Drugs Information 
Networks in Central Asia (NADIN) were identified as being potential partners and existing UN 
projects such as the Immediate Technical Assistance on control and prevention of drug and related 
organized crime in the Russian Federation, the Precursors Control in Central Asia and that 
concerning the establishment of a DCA in Tajikistan noted as foundations on which F23 could 
build. F23 also supported the ROCA Strategic Programme Framework and the Regional 
Programme for Afghanistan & Neighbouring Countries. 

The project document details an ambitious end-of-project situation as “/.../ the countries of the 
region will have national mechanisms for information collection, storage, analysis and exchange. 
These mechanisms will allow the drug law enforcement bodies in each country to collect and share 
police information with regard to illicit drug trafficking. The project will have provided both 
technical and organizational means for establishing procedures and networks for the collection 
and exchange of information{[...]”. 


Further it was recommended that the project became a part of the new UNODC “Programme for 
Central Asia 2015-2019: A partnership framework for impact related action in Central Asia 
(Kazakhstan, Kyrgyzstan, Tajikistan, Turkmenistan, Uzbekistan)”, to ensure the integrated 
programming approach which aims to deliver outcomes and outputs through sub-programmes 
rather than through standalone projects and initiatives and where a Regional Steering Committee 
was established to review and endorse strategic and operational priorities at the regional level. 


Evaluation Methodology 


The methods used to collect and analyse data 


The evaluation will be undertaken through a triangulation exercise of data stemming from desk 
review, structured interviews, as well as other sources to be established by the evaluation team. 
These could be primary data coming from questionnaires, surveys, or secondary data stemming 
from other entities. Quantitative as well qualitative data collection and analysis methods will be 
applied. Furthermore, the methodology will be gender-sensitive and inclusive and be further 
developed by the evaluation team as part of the inception report. 


BP) 


A desk review of relevant documents. These documents will include among others, the following: 
the project document, all project revision documents, semi-annual and annual project progress 
reports; 


Field mission to ROCA office in Tashkent consisting of: 


Briefing and individual interviews with Regional Representative, Senior Law 
Enforcement Advisor, Financial Officer. 


3. Field mission to Kazakhstan, Kyrgyzstan, Tajikistan and Uzbekistan consisting of: 


Individual interview with F23 Project Coordinator/Head of UNODC Tajikistan 
Programme-Office, Project Manager and briefings by UNODC staff members. 


Briefing by UNODC staff in the UNODC Programme-Offices in Kazakhstan, 
Kyrgyzstan, Tajikistan and Uzbekistan; 


Individual interviews with senior officials of project beneficiary agencies and other 
national counterparts, including officials from the Law Enforcement Agencies (LEA) 
of Kazakhstan, Kyrgyzstan, Tajikistan and Uzbekistan; 


Individual interviews with donor representatives; 


Site visits to Analytical Units of the LEA of Kazakhstan, Kyrgyzstan, Tajikistan and 
Uzbekistan and other sites for physical inspection and discussions both with 
beneficiaries and other people directly or indirectly affected by the project; 


Phone/Skype individual interviews with staff at UNODC HQ. 


The evaluation team will request further interviews, as needed. 


As part of the Inception Report, the evaluation team will develop a sound evaluation methodology 
as well as develop specific tools such as questionnaires and interview schedules, which will be 
reviewed and cleared by IEU prior to the field missions. 


Following completion of the missions, a draft evaluation report (in English) will be prepared, which 
will be sent to IEU for thorough review. The draft will be circulated to the Evaluation managers 
and to ROCA in sufficient time to allow for comments to be made and for any misunderstandings 
to be clarified. Final approval and clearance of the report will be done by IEU. The evaluators may 
choose to take the comments into account in producing the final report, for which they will be 
responsible. 


The UNODC Independent Evaluation Unit (IEU) will provide quality assurance and ensure 
compliance with the Norms and Standards of the United Nations Evaluation Group (UNEG) and 
UNODC guidelines* by providing comments on evaluation tools and methods, the draft report and 
clearance of the final report. 


The sources of data 


Interviews with CLPs and donors and relevant stakeholders outlined in Annex 3, as well as further 
interviews with stakeholders as proposed by the Evaluation team. Briefing and individual 


36 http://www.unodc.org/unodc/en/evaluation/about-projects-. html 
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interviews with International Regional Representative, Senior Law Enforcement Advisor, 
International Project Coordinator, Project Manager and Financial Officer. 


The evaluation will be undertaken through a triangulation exercise of data stemming from desk 
review, semi-structured interviews, field missions, questionnaires, observations and other sources 
to be established by the evaluator. 


The independent project evaluation is to be conducted following UNODC/IEU’s evaluation policy, 


norms, standards, guidelines and templates, (to be found on the JEU website, 
http://www.unodc.org/unodc/en/evaluation/index.html), as well as UNEG Norms and Standards. 
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Timeframe and Deliverables 


Duties Time frame Location | Deliverables 

Desk review and preparation of draft | 27 October — 9 Home base | Draft Inception report 

Inception Report November 2017 containing: preliminary 

(10 working days) findings of the desk review, 

refined evaluation questions, 
data collection instruments 
(including questionnaire and 
interview questions), 
sampling strategy, 
evaluation matrix and 
limitations to the evaluation 

Review and subsequent clearance of | 10—17 November Home base | Revised draft Inception 

draft Inception Report by the project | 2017 Report 

manager and IEU (can entail various | (3 working days) 

rounds of comments) 

Deliverable A: Final Inception By 17 November Final Inception report to 

Report in line with UNODC 2017 (13 working be cleared by IEU 

evaluation norms, standards, days) 

guidelines and templates 

Interviews with staff at UNODC HQ _ | 20 November - 1 UNODC/ Presentation of preliminary 

(phone or Skype interviews) and FO; | December 2017 HQ findings 

Evaluation mission: briefing, (10 working days) Countries/_ | Almaty, Astana, Kazakhstan; 

interviews; presentation of Cities Bishkek, Kyrgyzstan; 

preliminary findings Tashkent, Uzbekistan; and 
Dushanbe, Tajikistan 

Drafting of the evaluation report 4 - 18 December 2017 | Home base | Draft evaluation report 

(12 working days) 

Submission to Project Management 19-26 December 2017 

for review of factual errors and to 

IEU for review and comments (can 

entail various rounds of comments) 

Consideration of comments from the | 27 December 2017 — Home base | Revised draft evaluation 

project manager and incorporation of | 31 December 2017 report 

comments from IEU (3 working days) 

Deliverable B: Draft Evaluation By 31 December 2017 Draft evaluation report, to 

Report in line with UNODC (25 working days) be cleared by IEU 

evaluation norms, standards, 

guidelines and templates 

Draft evaluation report to be shared 15 — 29 January 2018 

with Core Learning Partners for 

comments 

Consideration of comments from 30 January 2018 Home base | Revised draft evaluation 

Core Learning Partners (1 working day) report 

Final review by IEU; incorporation of | 31 January — 9 Home base | Revised draft evaluation 

comments and finalization of report February 2018 report 

(can entail various rounds of (3 working days) 

comments) 


58 


ANNEXES 


Deliverable C: Final evaluation 
report; presentation of evaluation 
results 


By 9 February 2018 
(4 working days) 


Final evaluation report, to 
be cleared by IEU; 
presentation of evaluation 
results 


Project Management: Provide 
Management response (if needed) 
and finalise Evaluation Follow-up 
Plan in ProFi 


By 16 February 2018 


Final Evaluation Follow-up 
Plan to be cleared by IEU 


Project Management: Disseminate 
final evaluation report 


By 23 February 2018 


Final evaluation report 
disseminated 


Evaluation Team Composition 


Number of evaluators needed 


The final Independent Project Evaluation will be carried out by a team of two evaluators with one 
International Independent external lead evaluator and one national independent evaluator (team 
member) identified by UNODC through a competitive selection process, with logistical support 
provided by Project staff. 


The lead evaluator will have experience of evaluating technical assistance projects. The evaluator 
(team member) will be an expert in the area of criminal justice/law enforcement. 


The role of the Lead Evaluator 


Carry out the desk review; develop the inception report, including sample size and sampling 
technique; draft and finalize the inception report and evaluation methodology, incorporating 
relevant comments, in line with the norms, standards, guidelines and template on the IEU website 
http://www.unodc.org/unodc/en/evaluation/evaluation-step-by-step.html; implement quantitative 
as well as qualitative tools and analyze data; triangulate data and test rival explanations; ensure that 
all aspects of the terms of reference are fulfilled; draft an evaluation report in line with UNODC 
evaluation policy and the norms, standards, guidelines and template on the IEU website 
http://www.unodc.org/unodc/en/evaluation/evaluation-step-by-step.html; finalize the evaluation 
report on the basis of comments received; present the final evaluation findings and 
recommendations to stakeholders. 


Roles and Responsibilities of the Evaluator (team member) 
- assist the Lead Evaluator in all stages of the evaluation process, as per the respective TOR; 
- participate in selected missions. 


More details will be provided in the respective job descriptions in Annex I. 


Conflict of interest 
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According to UNODC rules, the evaluators must not have been involved in the design and/or 
implementation, supervision and coordination of and/or have benefited from _ the 
programme/project or theme under evaluation. 


Reference to job description detailing qualifications and responsibilities 


The lead evaluator should have the following qualifications and experience: 


¢ An academic degree and post graduate educational qualifications in social sciences, business 
administration or international development and/or graduation from a recognised criminal 
justice/law enforcement academy; 


¢ Substantial experience in evaluating technical assistance projects and/or programmes in 
international development and preferably regarding criminal justice/law enforcement (at least 7 
years professional experience); 


¢« Experience of having applied recognised quality management and assessment methodologies 
(such as the Balanced Scorecard or the Business Excellence Model of the EFQM) is desirable; 


¢ Familiarity with the criminal justice/law enforcement situation in the region will be an asset; 


* Technical knowledge of human rights and gender issues, including knowledge in women’s 
empowerment, gender mainstreaming, gender analysis and the related mandates within the UN 
system on gender and human rights; 


e« Fluency in spoken and written English required, with proven drafting skills, working 
knowledge of Russian is an asset. 


The lead evaluator will be responsible for drafting the evaluation report, ensuring the report meets 
the necessary standards and for submitting the drafts as described in a timely manner. 


The evaluator/team member should have the following qualifications and experience: 


An academic degree and post graduate educational qualifications in social sciences, business 
administration or international development and/or graduation from a recognised law enforcement 
academy 

A strong professional record in designing and leading independent evaluations 

Extensive knowledge of, and experience in applying, qualitative and quantitative evaluation 
methods 

Technical competence in the area of evaluation (at least 7 years) 

Previous work experience with undertaking project design, management and/or evaluation 
exercises with criminal justice projects / agencies, particularly those involving the police 
Experience of working on / with donor funded development projects in the Central Asian region 
Experience of working with UN agencies, and ideally with UNODC 

Excellent communication, facilitation and report writing / production skills 

Technical knowledge of human rights and gender based approach, including knowledge in 
women’s empowerment, gender mainstreaming, gender analysis and the related mandates within 
the UN system on gender and human rights 

Fluency in spoken and written English required, with proven drafting skills, working knowledge of 
Russian is highly desirable. 
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Experience and knowledge on gender equality and women’s empowerment, gender mainstreaming, 
gender analysis and the related mandates within the UN system — as well as experience and 
knowledge on human rights issues, the human rights based approach to programming, human rights 
analysis and related mandates within the UN system. 


Management of the Evaluation Process 
Roles and responsibilities of the Project Coordinator/Manager 


Management Arrangements 


The independent project evaluation will be carried out following UNODC’s evaluation policy and 
UNEG Norms and Standards. The evaluation team will work closely with UNODC’s Independent 
Evaluation Unit. 


Project Coordinator/Manager 
The Project Coordinator/Manager is responsible for: 


managing the evaluation, 
drafting and finalizing the ToR, 


selecting Core Learning Partners (representing a balance of men, women and other 
marginalised groups) and informing them of their role, 


recruiting evaluators following clearance by IEU, 


providing desk review materials (including data and information on men, women and other 
marginalised groups) to the evaluation team including the full TOR, 


reviewing the inception report as well as the evaluation methodology, 
liaising with the Core Learning Partners, 
reviewing the draft report for factual errors, 


developing an implementation plan for the evaluation recommendations as well as follow-up 
action (to be updated once per year), 


disseminate the final evaluation report and facilitate the presentation of evaluation results; 


The Project Coordinator/Manager will be in charge of providing logistical support to the 
evaluation team including arranging the field missions of the evaluation team, including but not 
limited to: 


All logistical arrangements for the travel of the consultants (including travel details; DSA- 
payments; transportation; independent interpretation; etc.) 


All logistical arrangement for the meetings/interviews/focus groups/etc., ensuring interview 
partners adequately represent men, women and other marginalised groups (including 


independent translator/interpreter if needed; set-up of meetings; arrangement of ad-hoc 
meetings as requested by the evaluation team; transportation from/to the interview venues; 
scheduling sufficient time for the interviews (around 45 minutes); ensuring that members 
of the evaluation team and the respective interviewees are present during the interviews; 
etc.) 


All logistical arrangements for the presentation of the evaluation results; 


Ensure timely payment of all fees/DSA/etc. (payments for the evaluators need to be released 
within 5 working days after the respective deliverable is cleared by IEU). 


For the field missions, the evaluation team liaises with the UNODC Regional/Field Offices and 
mentors as appropriate 


Roles and responsibilities of the evaluation stakeholders 


Members of the Core Learning Partnership (CLP) are selected by the project managers, 
representing a balance of men, women and other marginalised groups. The CLPs are the main 
stakeholders, i.e. a limited number of those deemed as particularly relevant to be involved 
throughout the evaluation process, i.e. in reviewing and commenting on the TOR and the evaluation 
questions, reviewing and commenting on the draft evaluation report, as well as facilitating the 
dissemination and application of the results and other follow-up action. Stakeholders include all 
those to be invited to participate in the interviews and surveys, including the CLPs. 


Roles and responsibilities of the Independent Evaluation Unit 


The Independent Evaluation Unit (IEU) provides mandatory normative tools, guidelines and 
templates to be used in the evaluation process. Please find the respective tools on the IEU web site 
http://www.unodc.org/unodc/en/evaluation/evaluation.html. 

TEU reviews and clears all steps and deliverables during the evaluation process: Terms of 
Reference; Selection of evaluator(s); Inception Report; Draft Evaluation Report; Final Evaluation 
Report; Evaluation Follow-up Plan. 


Payment Modalities 


The evaluator will be issued a consultancy contract and paid in accordance with UNODC rules and 
regulations. The contract is a legally binding document in which the evaluator agrees to complete 
the deliverables by the set deadlines. Payment is correlated to deliverables and three instalments 
are typically foreseen: 


. The first payment upon clearance of the Inception Report (in line with UNODC evaluation 
norms, standards, guidelines and templates) by IEU; 

° The second payment upon clearance of the Draft Evaluation Report (in line with UNODC 
norms, standards, evaluation guidelines and templates) by IEU; 

. The third and final payment (i.e. the remainder of the fee) only after completion of the 
respective tasks, receipt of the final report (in line with UNODC evaluation norms, standards, 
guidelines and templates) and clearance by IEU, as well as presentation of final evaluation findings 
and recommendations. 
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75 percent of the daily subsistence allowance and terminals is paid in advance before travelling. 
The balance is paid after the travel has taken place, upon presentation of boarding passes and the 
completed travel claim forms. 
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ANNEX I. TERMS OF REFERENCE FOR EVALUATOR 


Title: Independent Evaluator (Lead Evaluator) 

Organisational Section/Unit: UNODC Programme Office in Tajikistan 

Name and title of Supervisor: Amelia Hannaford, International Project Coordinator 

Duty Station or home-based: Home-based/with travel to Dushanbe Tajikistan; Bishkek, 
Kyrgyzstan; Almaty/Astana, Kazakhstan and Tashkent, 
Uzbekistan 

Proposed period: 27 October 2017— 23 February 2018 

Actual work time: 42 working days 

Fee Range: C 


Background of the assignment: 


The project’s objectives are to provide “a coherent and comprehensive system for intelligence, and 
information collection, analysis and exchange is established”. 


The project area of responsibility includes the five Central Asian countries (Kazakhstan, 
Kyrgyzstan, Tajikistan, Turkmenistan and Uzbekistan), Afghanistan and Azerbaijan. 


Combating illicit drug trafficking requires well-organized systems of information/data collection, 
processing and analysis, as well as the exchange of the final information product among agencies 
involved at national and regional levels. 


Project F23 has improved criminal intelligence analysis and information management capacities, 
and further developed mechanisms for collection, evaluation, exchange and dissemination of 
information between drug control agencies in the Central Asian region, Afghanistan and 
Azerbaijan. However the need to further developing a more sophisticated and proper mechanism 
for the collection, storage, exchange and analysis of information and data results is obvious and has 
been set as one of the priorities for the continuation of F23 project’s activities fully supported by 
the attending countries. 


At present, counter narcotic agencies, cooperating within the frame of project F23, collect 
information and data in accordance with local requirements and needs. 


The F23 “project philosophy” is to teach similar working methodologies and using same analysis 
software and database structures. Such approach will stimulate national and regional cooperation 
between agencies and encourage the sharing of information and analytical products between 
Intelligence analysts from participating countries. 


The project has installed modern intelligence systems and i2 analysis software in 18 counter 
narcotic agencies in the project area. Approximately 400 officers have been trained in crime 
analysis techniques at various levels. All relevant agencies have analysts working with 12 software 
who are able to create analytical reports and visualization charts in support of criminal 
investigations and operational activities. 
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It is necessary to continue developing a sustainable data and information collection capacity and 
root the Intelligence Led Policing implementation focusing on criminal intelligence analysis and 
information processing at counter narcotic Intel units. This exercise should also aim to identify 
priorities and a sound information gathering strategy for drug law enforcement bodies in each 
country. National drug agencies will be assisted in developing their information sharing and 
exchange system suitable to their objectives in combating illicit drug trafficking. 


Further details can be found in the Evaluation Terms of Reference. 


Purpose of the assignment: 


The purpose of this assignment is to conduct the final Independent Project Evaluation of RER/F23. 
The evaluation needs to assess all evaluation criteria of relevance, efficiency, partnerships and 
cooperation, human rights and gender mainstreaming of the project achievements/non- 
achievements, with a particular focus on effectiveness, impact and sustainability, as well as assess 
the implementation of recommendations from the mid-term Independent Project evaluation in 2012 
and as derive lessons learned, best practice and recommendations for future project interventions 
to ensure ownership, result-based orientation, cost-effectiveness and quality of the UNODC 
services. The evaluation-process needs to be fully independent and follow UNODC and UNEG 
evaluation norms and standards. The UNODC Independent Evaluation Unit is the clearing entity 
for all deliverables. 


Through this evaluation, UNODC ROCA will obtain an independent and objective assessment on 
the effectiveness of the activities conducted in the above specific areas and draw on 
recommendation and lessons learned to inform future programming. This evaluation will also 
suggest areas of improvement for the future. 


The results of this summative final evaluation are intended for use by the UNODC Regional Office 
for Central Asia, Project Team, beneficiary agencies and Donor Countries. In particular, it will 
serve as a reference source for the lessons learned from the UNODC Law Enforcement Programme, 
for its proper completion and integration with the necessary adjustments to the UNODC 
interventions in Central Asian region within the undergoing UNODC Regional Programme for 
Central Asian States 2015-2019 signed in May 2015 by the CA (Kazakhstan, Kyrgyzstan, 
Tajikistan, Turkmenistan, Uzbekistan) government representatives and based on the integrated 
programming approach which aims to deliver outcomes and outputs through sub-programmes 
rather than through standalone projects and initiatives. 


Specific tasks to be performed by the lead evaluator: 


Under the guidance of the Independent Evaluation Unit, the key responsibilities of the lead 
evaluator include (i) development of the evaluation design with detailed methods, tools and 
techniques, (ii) ensuring adherence to the UNEG Norms and Standards, UNODC evaluation norms, 
standards, guidelines and templates and the evaluation TOR, and (iii) ensuring that all deliverables 
are submitted in a timely and satisfactory manner and in line with the quality criteria checklist. 
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Expected tangible and measurable output(s)/deliverable(s): 


The lead evaluator will be responsible for the quality and timely submission of his/her specific 
deliverables, as specified below. All products should be well written, inclusive and have a clear 
analysis process. 


Draft inception report, containing: preliminary findings of the desk review, refined evaluation 
questions, data collection instruments (including questionnaire and interview questions), sampling 
strategy, evaluation matrix and limitations to the evaluation. 

Presentation of preliminary evaluation findings and recommendations to internal and external key 
stakeholders (if applicable). 

Draft evaluation report in line with UNODC evaluation norms, standards, guidelines and templates. 
Revised draft report based on comments received from the various consultative processes (IEU, 
internal and external). 

Final evaluation report, in line with UNODC evaluation norms, standards, guidelines and templates 
Final presentation of evaluation results to stakeholders. 


According to UNODC rules, the lead evaluator must not have been involved in the design and/or 
implementation, supervision and coordination of and/or have benefited from _ the 


programme/project or theme under evaluation. 


The lead evaluator shall respect the UNEG Ethical Guidelines. 


Dates and details of deliverables/payments: 


Deliverable | Output Working Days | To be accomplished 
by (date) 
A. Inception Report 13 working days_ | by 17 November 2017 
B. Draft Evaluation 25 working days | by 31 December 2017 
Report 
C. Final Evaluation 4 working days by 9 February 2018 
Report 


Payments will be made upon satisfactory completion and/or submission of outputs/deliverables. 


Indicators to evaluate the lead evaluator’s performance: 


Timely, satisfactory and high-quality delivery of the above mentioned outputs as assessed by IEU 
(in line with UNODC norms, standards, guidelines and templates as well as UNEG Standards and 


Norms). 
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Te Qualifications/expertise sought (required educational background, 
years of relevant work experience, other special skills or knowledge 
required): 


An academic degree and post graduate educational qualifications in social sciences, business 
administration or international development and/or graduation from a recognised law enforcement 
academy 

A strong professional record in designing and leading independent evaluations Extensive 
knowledge of, and experience in applying, qualitative and quantitative evaluation methods 
Technical competence in the area of evaluation (at least 7 years) 

Previous work experience with undertaking project design, management and/or evaluation 
exercises with criminal justice projects / agencies, particularly those involving the police 
Experience of working on / with donor funded development projects in the Central Asian region 
Experience of working with UN agencies, and ideally with UNODC 

Excellent communication, facilitation and report writing / production skills 

Technical knowledge of human rights and gender based approach, including knowledge in 
women’s empowerment, gender mainstreaming, gender analysis and the related mandates within 
the UN system on gender and human rights 

Fluency in spoken and written English required, with proven drafting skills, working knowledge of 
Russian is highly desirable. 

Experience and knowledge on gender equality and women’s empowerment, gender mainstreaming, 
gender analysis and the related mandates within the UN system — as well as experience and 
knowledge on human rights issues, the human rights based approach to programming, human rights 
analysis and related mandates within the UN system. 
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Title: Evaluator (team member) 


Organisational Section/Unit: UNODC Programme Office in Tajikistan 

Name and title of Supervisor: Amelia Hannaford, International Project Coordinator 

Duty Station or home-based: Home-based/with travel to Dushanbe Tajikistan; Bishkek, 
Kyrgyzstan; Almaty/Astana, Kazakhstan and Tashkent, 
Uzbekistan 

Proposed period: 27 October 2017— 23 February 2018 

Actual work time: 25 working days 

Fee Range: ¢ 


Background of the assignment: 


The project’s objectives are to provide “a coherent and comprehensive system for intelligence, and 
information collection, analysis and exchange is established”. 


The project area of responsibility includes the five Central Asian countries (Kazakhstan, 
Kyrgyzstan, Tajikistan, Turkmenistan and Uzbekistan), Afghanistan and Azerbaijan. 


Combating illicit drug trafficking requires well-organized systems of information/data collection, 
processing and analysis, as well as the exchange of the final information product among agencies 
involved at national and regional levels. 


Project F23 has improved criminal intelligence analysis and information management capacities, 
and further developed mechanisms for collection, evaluation, exchange and dissemination of 
information between drug control agencies in the Central Asian region, Afghanistan and 
Azerbaijan. However the need to further developing a more sophisticated and proper mechanism 
for the collection, storage, exchange and analysis of information and data results is obvious and has 
been set as one of the priorities for the continuation of F23 project’s activities fully supported by 
the attending countries. 


At present, counter narcotic agencies, cooperating within the frame of project F23, collect 
information and data in accordance with local requirements and needs. 


The F23 “project philosophy” is to teach similar working methodologies and using same analysis 
software and database structures. Such approach will stimulate national and regional cooperation 
between agencies and encourage the sharing of information and analytical products between 
Intelligence analysts from participating countries. 


The project has installed modern intelligence systems and i2 analysis software in 18 counter 
narcotic agencies in the project area. Approximately 400 officers have been trained in crime 
analysis techniques at various levels. All relevant agencies have analysts working with i2 software 
who are able to create analytical reports and visualization charts in support of criminal 
investigations and operational activities. 


It is necessary to continue developing a sustainable data and information collection capacity and 
root the Intelligence Led Policing implementation focusing on criminal intelligence analysis and 
information processing at counter narcotic Intel units. This exercise should also aim to identify 
priorities and a sound information gathering strategy for drug law enforcement bodies in each 
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country. National drug agencies will be assisted in developing their information sharing and 
exchange system suitable to their objectives in combating illicit drug trafficking. 


Further details can be found in the Evaluation Terms of Reference. 


Purpose of the assignment: 


The purpose of this assignment is to conduct the final Independent Project Evaluation of RER/F23. 
The evaluation needs is to assess all evaluation criteria of relevance, efficiency, partnerships and 
cooperation, human rights and gender mainstreaming of the project achievements/non- 
achievements, with a particular focus on effectiveness, impact and sustainability, as well as assess 
the implementation of recommendations from the mid-term Independent Project evaluation in 2012 
and as derive lessons learned, best practice and recommendations for future project interventions 
to ensure ownership, result-based orientation, cost-effectiveness and quality of the UNODC 
services. The evaluation-process needs to be fully independent and follow UNODC and UNEG 
evaluation norms and standards. The UNODC Independent Evaluation Unit is the clearing entity 
for all deliverables. Through this evaluation, UNODC ROCA will obtain an independent and 
objective assessment on the effectiveness of the activities conducted in the above specific areas and 
draw on recommendation and lessons learned to inform future programming. This evaluation will 
also suggest areas of improvement for the future. 


The results of this summative final evaluation are intended for use by the UNODC Regional Office 
for Central Asia, Project Team, beneficiary agencies and Donor Countries. In particular, it will 
serve as a reference source for the lessons learned from the UNODC Law Enforcement Programme, 
for its proper completion and integration with the necessary adjustments to the UNODC 
interventions in Central Asian region within the undergoing UNODC Regional Programme for 
Central Asian States 2015-2019 signed in May 2015 by the CA (Kazakhstan, Kyrgyzstan, 
Tajikistan, Turkmenistan, Uzbekistan) government representatives and based on the integrated 
programming approach which aims to deliver outcomes and outputs through sub-programmes 
rather than through standalone projects and initiatives. 


Specific tasks to be performed by the evaluator (team member): 
Under the guidance of the Independent Evaluation Unit, the key responsibilities of the evaluator 


(team member) is to collaborate with and support the lead evaluator with inputs to all deliverables 
in the evaluation process. 
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Expected tangible and measurable output(s)/deliverable(s): 


The evaluator (team member) will be responsible for collaborating with and support the lead 
evaluator with inputs to all the deliverables throughout the process and for the quality and timely 
submission of his/her specific deliverables, as specified below. All products should be well written, 
inclusive and have a clear analysis process. 


Draft inception report, containing: preliminary findings of the desk review, refined evaluation 
questions, data collection instruments (including questionnaire and interview questions), sampling 
strategy, evaluation matrix and limitations to the evaluation. 

Presentation of preliminary evaluation findings and recommendations to internal and external key 
stakeholders (if applicable). 

Draft evaluation report in line with UNODC evaluation norms, standards, guidelines and templates. 
Revised draft report based on comments received from the various consultative processes (IEU, 
internal and external). 

Final evaluation report, in line with UNODC evaluation norms, standards, guidelines and templates 
Final presentation of evaluation results to stakeholders. 


According to UNODC rules, the evaluator (team member) must not have been involved in the 
design and/or implementation, supervision and coordination of and/or have benefited from the 


programme/project or theme under evaluation. 


The evaluator (team member) shall respect the UNEG Ethical Guidelines. 


Dates and details of deliverables/payments: 


Deliverable | Output Working Days | To be accomplished 
by (date) 
A. Inception Report 8 working days by 17 November 2017 
B. Draft Evaluation 15 working days_ | by 31 December 2017 
Report 
C. Final Evaluation 2 working days by 9 February 2018 
Report 


Payments will be made upon satisfactory completion and/or submission of outputs/deliverables. 


Indicators to evaluate the evaluator’s performance: 
Timely, satisfactory and high-quality delivery of the above mentioned outputs as assessed by IEU 


(in line with UNODC norms, standards, guidelines and templates as well as UNEG Standards and 
Norms). 
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Te Qualifications/expertise sought (required educational background, 
years of relevant work experience, other special skills or knowledge 
required): 


An academic degree and post graduate educational qualifications in social sciences, business 
administration or international development and/or graduation from a recognised law enforcement 
academy 

A strong professional record in designing and leading independent evaluations 

Extensive knowledge of, and experience in applying, qualitative and quantitative evaluation 
methods 

Technical competence in the area of evaluation (at least 7 years) 

Previous work experience with undertaking project design, management and/or evaluation 
exercises with criminal justice projects / agencies, particularly those involving the police 
Experience of working on / with donor funded development projects in the Central Asian region 
Experience of working with UN agencies, and ideally with UNODC 

Excellent communication, facilitation and report writing / production skills 

Technical knowledge of human rights and gender based approach, including knowledge in 
women’s empowerment, gender mainstreaming, gender analysis and the related mandates within 
the UN system on gender and human rights 

Fluency in spoken and written English required, with proven drafting skills, working knowledge of 
Russian is highly desirable. 

Experience and knowledge on gender equality and women’s empowerment, gender mainstreaming, 
gender analysis and the related mandates within the UN system — as well as experience and 
knowledge on human rights issues, the human rights based approach to programming, human rights 
analysis and related mandates within the UN system. 
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interview guides 


Parts in grey shading were changed from the TOR. Parts in yellow shading are from the mid-term 
evaluation report. 


Evaluation questions 


Relevance*” 

Relevance is the extent to which the aid activity is continuously*® suited to the priorities and 
policies of the target group, recipient and donor. 

1. To what extent is / was the project relevant to the respective beneficiaries’ agencies and 
governments'39 needs and priorities? ... 

...To the donors' & UN? 

2. To what extent do the objectives, outcomes and outputs respond to present circumstances and 
stakeholder expectations? What would beneficiary agencies' drug law enforcement look like if it 
was just left to them? 


2012 Evaluation: Further regional development, involving the existing members of the project, 
possible new members and CARICC is undertaken by the use of bench-marking exercises that 
brings all beneficiaries together to compare national situations with agreed standards. Did this 
happen, did standards align with stakeholder interests on the one hand and project plans & activities 
on the other? 

2012 Evaluation: Needs analysis be conducted into the provision of new analytical and intelligence 
software packages; Did this help with alignment with actual needs? 


Efficiency 

Efficiency measures the outputs - qualitative and quantitative - in relation to the inputs. 

1. To what extent is the project implemented in the most efficient and cost-effective way compared 
to alternatives? Are the activities the most cost-effective way of achieving the outputs / outcomes / 
objective? 

2. Are there any good practices regarding efficiency, e.g. are certain aspects or arrangements of the 
project particularly efficient? 


Effectiveness 

Effectiveness is a measure of the extent to which an aid activity attains its objectives. 

1. To what degree did the project achieve its outcomes and objectives? What other major factors 
(apart from the project) helped or hindered? 

How good are beneficiary staffs skills relevant to the 3 Outcomes? How much do the 3 outcomes 
actually happen in live operations (proportion of time?) — What helps / hinders? what would this 
have been like without the project? 


37 UNODC guideline adds the “Design” criterion which is not mentioned in the ToR and is adequately covered 
under “relevance”. 

38 This word added from UNODC guideline 

39 Added from UNODC guideline 

40 Formulation changed in line with UNODC guideline (and the definition in the previous line) to clarify that the 
main interest is in what the programme itself achieved rather than what happened due to other reasons 
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How committed are stakeholder staff to the project Objectives? What helps / hinders them in this? 
2012 Evaluation: Comprehensive awareness programmes be conducted amongst beneficiary agency 
operational staff to ensure full understanding of intelligence-led law enforcement, its requirements 
and its benefits; - did they happen / help? 

2012 Evaluation: An international expert ‘team’/pairing be established to mentor and provide skills 
transfer in situ with beneficiaries and on a rolling basis amongst them; 

2. To what extent was the project management structure effective and allowed implementing the set 
objectives under the project? 1.e. UNODC management & within stakeholders 

4. To what extent did the project contribute to the Sustainable Development Goals? 

GENDER contribution to gender-specific SDGs in particular? 


Impact 

Impact is the positive and negative changes produced by a development intervention, directly or 
indirectly, intended or unintended". 

1. To what extent has the anticipated impact been reached by the project? 

2. Have there been any positive or negative unintended results? 

GENDER: specific impacts? 

HUMAN RIGHTS: specific impacts? 

2012 Evaluation: Continued support of CARICC via the work of the F23 project; 

2012 Evaluation: The inclusion of Customs Administrations in the project in a more formalised and 
embedded way. 

Sustainability 

Sustainability is concerned with measuring whether the benefits of an activity are likely to continue 
after donor funding has been withdrawn. 

1. What measures are in place to ensure future maintenance and repair of the facilities and 
equipment provided? 

2. To what extent are project interventions and impact” sustainable in the long term? Political will? 
What are the motivations of staff members? 

What is the most important change which will happen in drug trafficking and enforcement in the 
next 5 years? 

2012 Evaluation: Continued efforts to secure high level and donor support by UNODC, partners 
and beneficiary agencies . 

Increased donor visibility 

2012 Evaluation: Necessary IT updates be fulfilled by use of replaced UNODC equipment and 
renewed donor funding. Efforts should be made by beneficiaries for self-funding or direct outreach 
to potential donors; 

2012 Evaluation: Staff recruitment, rotation and retention issues be addressed, at senior level if 
necessary to ensure greater stability and sustainability with beneficiary agencies; 


Partnerships and cooperation 

The evaluation assesses the partnerships and cooperation established during the project® as well 
as their functioning and value. 

What is the nature of the relationship between the UNODC project and beneficiary agencies and 
BCs? (e.g. ,,donor-driven“, ,,bottom-up“ etc)? Whose project is it? 

1. To what extent were stakeholders properly engaged and informed? 

2. How was the project conducive to the development of agency partnerships at the bilateral and 
multilateral level? This includes a) between agencies within same country and b) internationally 
and c) with other national and international partners. 


41 As mentioned elsewhere, this implies clarifying what the intended impact actually was 
42 Added from UNODC guideline 
43 References to “project / programme” have been changed to “project” 
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3. To what extent has UNODC F-23 sought / established partnerships been-seught with national and 
international partners, including UN-agencies, UNODC-internally, public oversight bodies and civil 
society, etc.? 

2012 Evaluation: Delivery of assistance at regional level continues in the current model, using joint 
activities, in house training and expert meetings; 


Human rights 

The evaluation needs to assess the mainstreaming of human rights aspects throughout the project. 

1. To what extent are human rights considerations included in the project development and 
implementation? 

In addition, gender and human rights aspects of the other evaluation criteria will be included under 
their respective headings where relevant™. 


Gender 

The evaluation needs to assess the mainstreaming of gender aspects throughout the project. 

1. To what extent are gender considerations included in the project development and 
implementation? 

is gender-disaggregated data available? If so, analyse & interpret. 

2. To what extent did the project intervention contribute to fulfilment of the National Action Plan on 
implementation of UN Security Council Resolution 1325 on women, peace and security? 

To what extent did the project intervention contribute to introduction and implementation of 
measures aimed at increasing women and minority representation in the police? 


Lessons learned and best practices 

Lessons learned concern the learning experiences and insights that were gained throughout the 
project. 

1. What are the lessons learnt from the completed “Drug law enforcement systems for criminal 
intelligence collection, analysis and exchange” project? 

2. What are the best practices that could be applied in the future activities and similar projects? 
3. What are the necessary adjustments to the UNODC interventions in the region within the 
undergoing UNOD Programme for Central Asian States 2015-2019? 


DATA COLLECTION INSTRUMENTS 


Observation 


Observation of partner activities was carried out where possible during site visits to Analytical 
Units of the Law Enforcement Agencies of Kazakhstan, Kyrgyzstan, Tajikistan and Uzbekistan and 
other sites for physical inspection and discussions both with beneficiaries and other people directly 
or indirectly affected by the project. These visits were used to answer some additional questions as 
listed under the evaluation questions above, for example whether data collection and analysis are 
actually being carried out according to the project documentation. 


Deskwork 


Desk review of intervention records, desk review of published literature including comparison with 
standards. Project materials include the project documents, progress reports, outputs of the project. 


44 As suggested in UNODC guideline. Also, the ToR adds questions on recommendations on Human rights & 
Gender from the 2012 evaluation; but none were found. 
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Possible additional interviews with CSOs 


In this kind of project, especially one which has lasted over such a long period of time, it is normal 
to expect that there might be certain pressures on the agencies to answer our questions in ways 
which don't reflect the whole truth. The evaluation team suggested also talking to 1-3 NGOs in 1 or 
2 of the countries to get an additional source of information to ask them about this issue, to help 
better understand the information from the agencies. This was arranged. 
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ANNEX III. Evaluation matrix 


Methods have been chosen to best answer the evaluation questions and address the ToR. Each evaluation question below has been colour-coded for 
convenience to show the most important sources as follows: 

DW primarily answered by deskwork 

BA strong input from BAs 

DO strong input from donor 

UN strong input from UN 

OB observation 

This does mean a lot of evaluation questions are to be answered by beneficiary agencies, so some of those marked here may be skipped for individual 
BAs in order to focus on the questions an individual respondent is best placed to answer. 

Clarify whether it might make sense to ask some or all of the appropriate questions in advance in written form. 

Other sources may of course also provide data for questions for which they are not explicitly marked — for example, deskwork could turn up additional 
data for any of the questions. 

It would be possible to break down these questions further into more detailed sub-questions. However as the evaluation report is to have maximum 25-30 
pages, breaking the questions down further is not recommended. 
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Relevance” 


1. To what extent is / was the project relevant to the respective Triangulatio | Ask contextualised | BA See 
beneficiaries’ agencies and governments'46 needs and priorities? | n between all | question comme 
sources Prompt based on nts in 
presentation of section 
draft Theory of on 
Change Sampli 
Compare project DW ng 
docs with National 
strategies 
... To the donors' & UN? Triangulatio | Ask contextualised | UN DO 
n between all | question 
sources Prompt based on 
presentation of 
draft Theory of 
Change 


Compare project 
docs with strategy 
docs, UNDAF, etc 


2. To what extent do the objectives, outcomes and outputs Triangulatio | Ask contextualised | BA 
respond to present circumstances and stakeholder expectations? | n between all | question 
What would BAs' drug law enforcement look like if it was just sources Prompt based on 
left to them? presentation of 
draft Theory of 
Change 


Compare project DW 
docs with strategy 
docs and 
background 


45 UNODC guideline adds the “Design” criterion which is not mentioned in the ToR and is adequately covered under “relevance”. 
46 Added from UNODC guideline 
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information e.g. 
INL documentation 


2012 Evaluation: Further regional development, involving the Triangulatio | Ask direct question | BA 
existing members of the project, possible new members and n between all 
CARICC is undertaken by the use of bench-marking exercises sources Compare project 
that brings all beneficiaries together to compare national plan with project DW 
situations with agreed standards. Did this happen, did standards monitoring docs, 
align with stakeholder interests on the one hand and project financials. Identify 
plans & activities on the other? what were the 
“agreed standards” 
2012 Evaluation: Needs analysis be conducted into the provision | Triangulatio | Ask direct question | BA 
of new analytical and intelligence software packages; Did this n between all 
help with alignment with actual needs? sources Compare project 
plan with project DW 
monitoring docs. 
Efficiency 
1. To what extent is the project implemented in the most Triangulatio | Ask about specific DW (reference to 
efficient and cost-effective way compared to alternatives? Are n between all | items from budget budgets)BA DO 
the activities the most cost-effective way of achieving the sources Ask direct question 


outputs / outcomes / objective? 


Ask for specific 
examples 


e.g. verify presence 
of specific 
hardware & 
software 


OB (observation of 
waste) 


Try to 
insist 
on 
observ 
ation 
in each 
active 
unit 
visited 
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howev 
er may 
not be 
meanin 
gful / 
possibl 
ein 
some 
occasi 
ons 
2. Are there any good practices regarding efficiency, e.g. are Triangulatio | Ask direct question | UN BA 
certain aspects or arrangements of the project particularly n between all | Ask for specific 
efficient? sources examples 
Locate good 
examples from 
project monitoring 
data 
Effectiveness 
1. To what degree did the project achieve its outcomes and Triangulatio | (reference to DW 
objectives*”? What other major factors (apart from the project) n between all | project monitoring 
helped or hindered? sources documentation, 
financials) 
Search for other 
relevant contextual 
factors in other See 
agency reports e.g. BA UN comme 
INL nts on 
Direct question, observ 
also ask about 


47 Formulation changed in line with UNODC guideline (and the definition in the previous line) to clarify that the main interest is in what the programme itself achieved rather 
than what happened due to other reasons 
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monitoring results, 
see above 


Observation, do 
observed activities 
match project plan? 


OB 


How good are beneficiary staffs skills relevant to the 3 
Outcomes? How much do the 3 outcomes actually happen in live 
operations (proportion of time?) — What helps / hinders? what 
would this have been like without the project? 

GENDER Gender disaggregation of staff & skills? 


Triangulatio 
n between all 
sources 


Ask contextualised 
question and 
compare answers 
from BA and UN 
sources 

Clarify if possible 
to ask for data on 
the 3 outcomes 
from agencies e.g. 
proportion of time 
spent on them 
Direct observation 
if possible, focus on 
actual skills and use 


BA UN 


OB 


How committed are stakeholder staff to the project Objectives? 
What helps / hinders them in this? 2012 Evaluation: 
Comprehensive awareness programmes be conducted amongst 
BA operational staff to ensure full understanding of intelligence- 
led law enforcement, its requirements and its benefits; - did they 
happen / help? 


Triangulatio 
n between all 
sources 


Ask indirect 
question 

Ask direct question 
& verify from 
monitoring docs, 
financials 

Deduce level of 
commitment from 
respondents’ 
answers and 
attitude 


BA 


OB 


Indirect 
question e.g. 
what do you 
think about 
others, what 
are main 
motivations / 
rewards in 
this job 
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2012 Evaluation: An international expert ‘team’/pairing be Triangulatio | Direct question — BA 
established to mentor and provide skills transfer in situ with n between all | did this happen. 
beneficiaries and on a rolling basis amongst them; sources Ask for evidence. 
Verify from project DW 
monitoring docs, 
financials 
2. To what extent was the project management structure Triangulatio | Indirect question— | BADO 
effective and allowed implementing the set objectives under the | n; In ask about problems 
project? i.e. UNODC management & within stakeholders particular, & responses; ask 
compare about others. 
answers from 
both 
respondent 
groups 
4. To what extent did the project contribute to the Sustainable Triangulatio | Plausibility analysis DWiincluding other 
Development Goals? n between all | on base of reports about 
sources outcomes reported general linkages 
GENDER contribution to gender-specific SDGs in particular? in annual reports & | between project 
impact analysis, Outcomes / 
below Objectives and 
SDGs 
Impact 
1. To what extent has the anticipated impact been reached by the | Triangulatio | Direct question, BA UN DO 
project? n between all | after using DW and 
sources interview to try to 
clarify what the DW 
intended impact 
was 
2. Have there been any positive or negative unintended results? Triangulatio | Ask for other BA UN DO 


GENDER: specific impacts? 
HUMAN RIGHTS: specific impacts? 


n between all 
sources, also 
using 


examples 
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prompts 
suggested by 
other sources 


Identify other 
examples in project 
M&E docs and 
national level 


Analysis of how 
sustainable outputs 


DW — project 


documentation 


reporting 
2012 Evaluation: Continued support of CARICC via the work of | Triangulatio | Direct question; did | BAUN 
the F23 project; n between all | it happen? 
sources 
Verify in project DW 
M&E, financials 
documentation 
2012 Evaluation: The inclusion of Customs Administrations in Triangulatio | Direct question; did | BAUN 
the project in a more formalised and embedded way. n between all | it happen? 
sources In particular, direct 
question to CAs 
(UZB, TAJ) DW 
Verify in project 
M&E, financials 
Sustainability 
1. What measures are in place to ensure future maintenance and | Triangulatio | Direct question BA DO 
repair of the facilities and equipment provided? n between all | Ask for examples 
sources of existing 
maintenance DW 
2. To what extent are project interventions and impact*® Triangulatio | Direct question. BA UN DO Motivation — 
sustainable in the long term? Political will? What are the n between all | Indirect questions see above 
motivations of staff members? sources about motivation. 


4s Added from UNODC guideline 
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and outcomes have 
been to date 


within same agency 


DW — hopefully on 
basis of staff data 
from BAs 


What is the most important change which will happen in drug Triangulatio | Direct question BA UN 
trafficking and enforcement in the next 5 years? n between all 
sources DW — background 
Interpret the information e.g. 
responses from INL reports 
from the 
perspective 
of 
sustainability 
2012 Evaluation: Continued efforts to secure high level and Triangulatio | Direct questions, BA DO 
donor support by UNODC, partners and beneficiary agencies n between all | ask for examples & 
(BA)s. sources success DW — project 
Increased donor visibility activity reports 
Look for examples | OB 
of donor visibility 
2012 Evaluation: Necessary IT updates be fulfilled by use of Triangulatio | As above BA DO 
replaced UNODC equipment and renewed donor funding. n between all 
Efforts should be made by beneficiaries for self-funding or sources DW — project 
direct outreach to potential donors; activity reports & 
Is equipment in financials 
place? OB 
2012 Evaluation: Staff recruitment, rotation and retention issues | Triangulatio | Direct question—in | BA Could as for 
be addressed, at senior level if necessary to ensure greater n between all | comparison with this in 
stability and sustainability with BAs; sources other functions advance 
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Partnerships and cooperation 


What is the nature of the relationship between the UNODC Triangulatio | Interpret answers to | BAUN 

project and BAs and BCs? (e.g. ,,donor-driven“, ,,bottom-up“ n between all | other questions; 

etc)? Whose project is it? sources also ask directly 

1. To what extent were stakeholders properly engaged and Triangulatio | As above BA DO 

informed? n between all 

sources 

2. How was the project conducive to the development of agency | Triangulatio | Ask directly; ask BA 

partnerships at the bilateral and multilateral level? This includes | n between all | for examples 

a) between agencies within same country and b) internationally sources DW - 

and c) with other national and international partners. documentation of 
interagency links 
from BAs? 

3. To what extent has UNODC F-23 sought / established Triangulatio | As above UN DO 

partnerships been-seught with national and international n between all 

partners, including UN-agencies, UNODC-internally, public sources DW - 

oversight bodies and civil society, etc.? documentation of 
interagency links 

2012 Evaluation: Delivery of assistance at regional level Triangulatio | Ask directly BA UN 

continues in the current model, using joint activities, in house n between all 

training and expert meetings; sources Desk work DW - 
documentation of 
regional assistance 

Human rights 

1. To what extent are human rights considerations included in Triangulatio | Direct question; ask | UN DO 

the project development and implementation? n between all | for examples 

sources DW - 


documentation of 
interagency links 
from BAs 
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In addition, gender and human rights aspects of the other 
evaluation criteria will be included under their respective 
headings where relevant”. 


See also questions 
marked GENDER 
and HUMAN 
RIGHTS above 


questions looking 
for lessons using 
evaluative 
reasoning 


Gender 
1. To what extent are gender considerations included in the Triangulatio | Direct question; ask | UN DO 
project development and implementation? n between all | for examples 
sources 
pw - project docs 

is gender-disaggregated data available? If so, analyse & Desk work 
interpret. 
2. To what extent did the project intervention contribute to Triangulatio | Direct question; ask | UN 
fulfilment of the National Action Plan on implementation of UN | n between all | for examples DW. - national docs 
Security Council Resolution 1325 on women, peace and sources & strategy 
security? 
To what extent did the project intervention contribute to Triangulatio | Direct question; ask | UN 
introduction and implementation of measures aimed at n between all | for examples DW. - national docs 
increasing women and minority representation in the police? sources & strategy 
Lessons learned and best practices 
1. What are the lessons learnt from the completed “Drug law Triangulatio | Direct question BA UN DO Note that 
enforcement systems for criminal intelligence collection, n between all Lessons 
analysis and exchange” project? sources Also interpret Learned is 

answers from other interpreted 


here as what 
was actually 
learned 


49 As suggested in UNODC guideline. Also, the ToR adds questions on recommendations on Human rights & Gender from the 2012 evaluation; but none were found. 
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2. What are the best practices that could be applied in the future | Triangulatio | Direct question BA UN DO 
activities and similar projects? n between all | Also interpret 
sources answers from other 


questions looking 
for lessons using 


evaluative 

reasoning 
3. What are the necessary adjustments to the UNODC Triangulatio | Direct question UN 
interventions in the region within the undergoing UNODC n between all 
Regional Programme for Central Asian States 2015-2019? sources Also interpret 


answers from other 
questions looking 
for lessons using 
evaluative 
reasoning 
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ANNEX IV. Desk review list 


UNODC documents 


UNODC Strategic documents 
UNODC Medium-term Strategy 2012-2015 


UNODC Strategic Framework 2014-2015 


Strategic Outline for Central Asia and Southern Caucasus 2012 — 2015; A comprehensive approach 
to implement the UNODC drug and crime mandate 


UNODC Programme for Central Asia 2015-2019: A Partnership Framework for Impact Related 
Action in CA 


Project documents 
Project Document (07.06.2001) 


Financial and budget documents 


Project revision documents 
Project Revision document (13.02.2003) 


Project Revision Document (06.10.2005) 
Project Revision Document (14.05.2007) 
Project Revision Document (04.08.2009) 
Project Revision Document (22.12.2010) 
Project Revision Document (22.06.2012) 
Project Revision Document (22.07.2014) 


Project annual and semi-annual progress reports 
Annual Project Progress Reports (2012, 2013, 2014, 2015 and 2016) 


ToR of the final Independent Project Evaluation 


Mid-term evaluation of TD/RER/F23 (2012) 


UNODC Guidance documents 
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UNODC Position Paper on Human Rights (2011) 

Guidance Note on Gender Mainstreaming in UNODC (2013) 
UNODC evaluation guidelines, templates, handbook, policy 
UNODC Inception Report Guidelines and Template 


UNODC Evaluation Report Guidelines and Template 


UNODC and the SDGs Brochure 


2030 Agenda for Sustainable Development (SDGS) 


Other external documents 


UNEG: Integrating human rights and gender equality in evaluation 
UNDAF s for all four countries 


National Development Plans for Uzbekistan, Kazakhstan, Kyrgyzstan and National Drug Strategy 
for Tajikistan 


INL International Narcotics Control Strategy Report 2017 vol. 1 & 2 


USAID Central Asia Regional Development Cooperation Strategy 
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ANNEX V. List of persons contacted during the 
evaluation 


Organisation Type of stakeholders0 Sex disaggregated Country 
data 

UNODC 

Tajikistan & F23 Female:2 

office Project management Male: Tajikistan 
Female: 

INL Tajikistan Donor Male: 1 Tajikistan 
Female: 

Spin Plus Civil Society Male: 1 Tajikistan 

Ministry of Female: 

Internal Affairs beneficiary agency Male: 3 Tajikistan 
Female: 

AFEW Civil Society Male: 2 Tajikistan 


Drug Control 


Agency, Female: 1 

Analytical Unit = beneficiary agency Male: 5 Tajikistan 

Ministry of Female: 

Foreign Affairs © Other Govt.Agency Male: 2 Tajikistan 

UNODC Female: 1 

Kazakhstan Project management Male: Kazakhstan (Astana) 
Female: 1 

INL Kazakhstan Donor Male: Kazakhstan (Astana) 

Agency for Civil 

Service and Anti- Female: 

Corruption beneficiary agency Male: 4 Kazakhstan (Astana) 

Ministry of Female: 

Internal Affairs, | beneficiary agency Male: 4 Kazakhstan (Astana) 


50 This could be e.g. Civil Society Organisation; Project/Programme implementer; Government recipient; Donor; 
Academia/Research institute; etc. 
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Counter- 
Narcotics 
Department 


Ministry of 
Foreign Affairs 


UNODC 
Tajikistan & F23 
office (2nd 
meeting) 


CARICC 


MOI Drug 
Control Service 


UNODC 
Kyrgyzstan 


MOI Operational 
Analysis 
Department 


National Center 
on Drug Control; 
Ministry of 
Foreign Affairs; 
Ministry of 
Internal Affairs; 
State Custom 
Committee 


ROCA Regional 
Representative 


ROCA 
Monitoring & 
Evaluation 
Officer 


ROCA Financial 
Officer 


F23 Project 
Manager 


Total 


Other Govt. Agency 


Project management 


beneficiary agency 


beneficiary agency 


Project Management 


beneficiary agency 


beneficiary agency 


Project Management 


Project Management 


Project Management 


Project Management 


Female: 
Male: 1 


Female: 
Male: 


Female: 
Male: 4 


Female: 
Male: 2 


Female: 
Male: 1 


Female: 
Male: 3 


Female: 
Male: 5 


Female: 
Male: 0 


Female: 
Male: 1 


Female: 
Male: 1 


Female: 
Male: 1 


Female: 
Male: 43 


= 


av 


0 


1 


10 
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Kazakhstan (Astana) 


Kazakhstan (Almaty) 


Kazakhstan (Almaty) 


Kyrgyzstan 


Kyrgyzstan 


Kyrgyzstan 


Uzbekistan 


Uzbekistan 


Uzbekistan 


Uzbekistan 


Uzbekistan 


ANNEXES 


ANNEX VI. = Sampling strategy 


The project coverage is 9 countries, however since 2012 F23 had available funds only for the 
Central Asia states received from INL USA. Out of 5 Central Asia states Turkmenistan did not sign 
the project extension for internal reasons. As a result since 2012 the project was implemented only 
in these four countries, therefore at the country level the sample is a so-called census, i.e. all are 
covered. 


So agency selection is only relevant in terms of selecting agencies within countries. 


Statistical sample-size calculations are not relevant with such a small sample especially as nearly 
all the data is qualitative rather than quantitative. However representativeness is an important 
question. 


Work was carried out with the F23 office to clarify how typical the pre-selected agencies are; if 
they are not typical, what is different about those who are not covered. Conclusion was that 
(understandably) the sample covers the agencies with better cooperation, so this was taken into 
account in interpreting the results — see footnote on p. 8. 
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ANNEX VII. = CURRENT LOGFRAME 


Outcome 1 


Specialized law enforcement personnel use enhanced management of information 
gathering and exchange systems. 


Unchanged 

Output 1.1 

Necessary training for the use of new software is provided. 
Activities 


1.1.1 Provision of Intelligence analysis training for the LE analysts of counter 
narcotics LEA of project member states. 


Output 1.2 

Train the trainer approach to develop a self-sustaining competence. 
Reformulated 

Activity 


1.2.1 To organize train-the-trainers courses to develop a self - sustaining national 
competence within each agency (Reformulated). 


Activity 


1.2.2 To organize peer- to—peer seminars for the LE analysts of the LE agencies at 
the regional level (New). 


Outcome 2 


Law enforcement personnel use enhanced analytical capacity in operational work. 


Reformulated 
Output 2.1 


Conduct an assessment of analytical and intelligence led policing capacity in respond 
to the existing needs. 


Reformulated 
Activities 


2.1.1 Assess the analytical capacity of the Intelligence law enforcement structures in 
respond to the existing needs (Reformulated); 
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2.1.2 Work plan for realistic and feasible implementations of criminal intelligence 
analysis technique and methodologies prepared (New); 


2.1.3 Intelligence analysts producing analytical reports as a complement to criminal 
investigations and operations and to the senior leadership as a resource planning and 
strategic analysis support (Reformulated). 


Output 2.2 


Enhance efforts of the beneficiaries for self-funding or direct outreach to potential 
donors. 


New 
Activity 


2.2.1 Regular meetings organized with the beneficiaries and donors to discuss 
funding possibilities and self-sustainability issues (New). 


Outcome 3 
Law enforcement personnel increase use of intelligence-led policing tools in 
operational work. 


Output 3.1 


Necessary IT equipment (hardware and software) provided to meet existing 
challenges. 


Reformulated 


Activity 


3.1.1 The IT equipment (hardware and software) provided/upgraded based on the 
individual requests from the LE analytical units/departments (Reformulated); 

3.1.2 To create and install intelligence database for storage and analysis of collected 
information related to the law enforcement structures (Reformulated). 


Output 3.2 


Organize regular semi-annual regional meetings involving participating countries, 
with a view to increasing knowledge and strengthening LE (analytical) cooperation 
and information exchange. 


Reformulated 


Activity 

3.2.1 To conduct regular experts meetings together with CARICC with a view to 
continue developing a strategy and implementation plan how to root and establish 
sustainable information exchange channels between CARICC and the counter 
narcotic law enforcement agencies in the region. 
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ANNEX VIII. COMMENTS ON LOGICAL 
FRAMEWORK 


The project documents including logical framework were discussed and cleared by the relevant 
governments’ agencies. Below, the project logical framework is examined to explicate more 
precisely what the project was actually aiming to do in order to be able to evaluate its actual 
achievement against those aims. This step is essential particularly because the evaluation is 
supposed to be summative in nature5!. 


Objective 


The project Objective is: “A coherent and comprehensive system for intelligence, information and 
information collection, analysis and exchange is established.” This is quite a limited (though not 
necessarily easy-to-achieve) aim as it deals only with the establishment of a system (i.e. a network, 
agencies and functional tools) and not with whether that system is used or what the positive 
consequences of that should be. Indeed, the original project Objective was “To increase interdiction 
and the crime control effectiveness of drug law enforcement structures...” which was then reduced 
to “To increase crime control effectiveness ...” and again, whereas the present Objective is reduced 
again only to the establishment of systems — a much less ambitious but perhaps more realistic 
Objective. But does the establishment of a system also imply that it is actually used or even that 
such use is to have positive consequences? 


On the one hand it seems not, because the current Indicator for this is “Each project country has at 
least operational analysis unit52 with functional and comprehensive database as a support tool for 
the ongoing investigations.” — and indeed this is a meaningful aim as this was not the case 
originally; but no baseline is defined and obviously it is difficult to define “functional and 
comprehensive”. 


On the other hand, all three Outcomes below this Objective do clearly mention use, see below. So 
it seems that the Objective is intended as a “definitional Objective” i.e. the three Outcomes define 
it, rather than cause it. This means that no separate indicator is needed for the Objective, as it is 
deemed as being achieved when the Outcomes are achieved. And it suggests that the Objective is 
indeed to be understood as implying not just the establishment but also the use of systems. 


Outcomes 


The three Outcomes are as follows (slightly abbreviated). The phrases in *italics* are presumably 
intended to carry express the intended changes, the difference made by the project. 


51 Scriven, M. (1991). Beyond formative and summative evaluation. 
52 Presumably this should read “Each project country has at least ONE operational analysis unit” 
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Specialized law enforcement personnel use *enhanced* management of information- 
gathering and exchange systems. 


Law enforcement personnel use *enhanced* analytical capacity in operational work 


Law enforcement personnel *increased* use of intelligence-led policing in operational 
work [emphasis added] 


So each Outcome focuses on use. This is admirable successful and sustainable technical assistance 
usually depends on behaviour change of key players. 


The Outcomes also imply that (at least in most cases) the groundwork has already been done 
(helping to establish specific analytical units with trained staff and relevant hardware and software 
in the initial project phases) and the focus is now on increasing / improving actual use. This is 
consistent with the findings of the mid-term evaluationS3. 


The Objective and Outcomes nowhere define the desired extent / scale of the project — do they refer 
to one beneficiary agency, or all of them, or to individual countries? Indeed the indicator for the 
Objective (“Each project country has at least operational analysis unit [sic; and emphasis added] 
with functional and comprehensive database as a support tool for the ongoing investigations”) is 
hard to understand — does this mean each country has at least one such agency?— is it enough to 
have just one agency functioning in each country? If not, which different agencies should be 
involved? 


Does the logical framework imply or state which kind of crime is to be targeted? Trafficking only 
of narcotics and only by organised transnational criminal groups would be the narrowest 
interpretation consistent with the project title; but the Objectives and Outcomes leave broader 
interpretations open. 


Finally, does the logical framework imply or state what kind of data exchange between agencies is 
intended? In particular, is data to be exchanged within or between countries? The narratives in the 
project documents imply between countries. E.g. “Combating illicit drug trafficking requires 


well-organized systems of information/data collection, processing and analysis, 


the exchange of the final information product among agencies involved at national and regional 
levels” 


But in the logframe itself, only the indicator (but not text) for Outcome 3 mentions international 
collaboration. 


53 From 2011 Annual Report: “The project has now installed modern intelligence systems and i2 analysis software 


in 17 counter narcotic agencies throughout Central Asia Afghanistan and Azerbaijan. Approximately 180 officers 
have been trained in crime analysis techniques at various levels. All relevant agencies have analysts working with 
i2 software who are able to create analytical reports and visualization charts in support of operational activity.” 
This differs only slightly from the parallel statement in the 2017 semi-annual report, implying that the new focus 


is on strengthening existing achievements. 
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The suggested minimum and additional success dimensions are as follows. 


Essential success dimensions: Within at least one agency within each participating country, with 
respect to trafficking of narcotics, the project must increase/enhance the three Outcomes: 1) 
information gathering and 2) analysis, and 3) this is to be increasingly intelligence -led. 
“Intelligence-led” means in particular that the work of agencies is not only responding to crimes 
but using intelligence to understand and anticipate crime in a pro-active way, also impacting 
resource allocations4; so the overall success of the project is to be judged against these three 


outcomes. There are also additional aims (“nice to have”) which are implied at various points in 
the project documentation but not so consistently that they can be judged to be binding: 


Additional success dimensions: 

progress on these dimensions in more than one agency in each country 

application a) in particular to transnational narcotics trafficking and b) more generally to other 
organised crimes 

increased exchange of intelligence between agencies within one country 

increased exchange of intelligence between countries 

This still leaves open which countries are “the participating countries” — this phrase is used in the 
Logframe but the only list of countries is the Project Location which includes the full list - Central 
Asian States, Afghanistan, Armenia, Azerbaijan and Georgia. It is acknowledged that funding was 
only in fact available for the four countries covered in the ToR for this evaluation mission, so these 
four countries will be considered to exhaust the participating countries for the purposes of this 
evaluation, although wider networking includes also those countries which were previously 
involved, coordinated by the Central Asian Regional Information and Coordination Centre 
(CARICC)*. 


The question remains how much improvement is required or implied on these essential and 
additional success dimensions. Inspection of the indicators, targets and baselines in the logical 
framework does not help as they are somewhat difficult to understand and not very consistent5°. 


So it will be assumed that any difference made to these dimensions should be counted as success57. 


Project documentation implies that all these dimensions are synergistic, i.e. improvement on any 
of them is likely to contribute to improving the whole system: an overall, intelligence-led and 
intelligence-sharing, multiple-agency, international counter-narcotics system: “Combating illicit 
drug trafficking requires well-organized systems of information/data collection, processing and 


54 Ratcliffe, J. H. (2016). Intelligence-led policing. Routledge. 

55 However this is not standard practice; normally if the coverage of a project is reduced compared to the original 
plan, this is noted within a project revision. 

56 In particular, Outcome 1: Baseline refers to skills, Target refers to trainings delivered and Outcome refers to 
use. Baseline and Target for Outcome 2 refer to Output 2.1 (conducting needs analyses) rather than to use 
of analytical capacity. Outcome 3: Baseline and Target refer to expert meetings whereas the Outcome refers 
to use of intelligence-led policing 

57 In particular, project reports from previous years make it clear that in some cases there may be a tendency for 
some achievements to regress over time without additional input. This means that the baseline level of a 
criterion or indicator is not a fair estimate of the counterfactual 
(http://www. betterevaluation.org/en/plan/understandcauses/compare_results_to_ counterfactual). For 
example merely maintaining a system or level of use in a particular agency over time may in some 
circumstances represent a significant achievement; and raising it even a little may represent an even bigger 
one. This is why the word “difference” rather than “change” has been used above. 
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analysis, as well as the exchange of the final information product among agencies involved at 
national and regional levels’’58 


Figure 5. A graphical representation of the project logical framework from the latest 
Revision (7), with adjustments* 


Outputs 


Output 1.1 
Necessary training for the use of new software is 
provided. 


Output 1.2 
Train the trainer approach to develop a 
self-sustaining competence. 


Output 2.1 

Conduct an assessment of analytical and 
intelligence led policing capacity in respond to the 
existing needs. 


Output 2.2 
Enhance efforts of the beneficiaries for self-funding 
or direct outreach to potential donors. 


Output 3.1 
Necessary IT equipment (hardware and software) 


Outcome 1 

Specialized law enforcement personnel use 
enhanced management of information gathering 
and exchange systems. 


Outcome 2 
Law enforcement personnel use enhanced 
analytical capacity in operational work 


Outcome 3 
Law enforcement personnel increase use of 
intelligence-led policing tools in operational work 


wi; Project Objective: 

: A coherent and comprehensive system for : 
: intelligence, information and information collection, : 
: analysis and exchange is established 4 
: Each project country has at least operational 

: analysis unit with functional and comprehensive 
: database as a support tool for the ongoing 
vestigations. 


provided to meet existing challenges. 


Output 3.2 

Organize regular semi-annual regional meetings 
involving participating countries, with a view to 
increasing knowledge and strengthening LE 
(analytical) cooperation and information exchange. 


*All Outputs potentially contribute to each Outcome; dotted lines to and around Objective show it is defined in terms of 
the Outcomes rather than caused by them 


Finally, how do the project Outputs contribute to the Outcomes? Most of the Outputs contribute to 
more than one or even to all three Outcomes59. So for example Output 1.1, training on software, 


helps with all three Outcomes, not just Outcome | as implied in the logframe. 


Perhaps the logframe can be better understood as in Figure 2. This reproduces the Logical 
framework exactly except for two changes: 1) the Outputs collectively lead to the Outcomes (so 
each and every Output potentially contributes to Outcomes 1, 2, and 3); and 2) the Objective is 
shown with dotted lines because it is defined as a summary of the Outcomes rather than being an 
independent variable which is caused by them. 


Deeper understanding of how the project is intended to work depends on understanding factors not 
shown in this diagram or explicitly mentioned in the logframe. It is quite normal that project 


58 Original PROJECT DOCUMENT: AD/RER/00/F23. 
59 With the kind of hierarchical logframe format used at UNODC it is difficult to show an Output contributing to 
more than one Outcome 
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logframes omit some of these additional factors, perhaps for the sake of simplicity. But for this 
evaluation it will be useful to discuss them. One omission is skills which form the crucial link 
between Outputs and Outcomes — for example, an officer attends a training course (Output), 
develops skills (missing link) with which they are able to use the systems (Outcomes). 


Another omission is the many other important inputs into the mechanism which are provided by 
others, in particular national governments which of course provide the majority of inputs into 
intelligence systems (staffing, premises, etc) and individuals (whose performance depends on their 
own motivation etc). 
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ANNEX IX. = MORE DETAILS ON THE 
RECOMMENDATION ON MONITORING 
AND EVALUATION 


This text follows on from the corresponding outline recommendation on p. 37. 


It has proved difficult in F23 to predict in advance which areas and topics will prove most possible 
and most fruitful, so the target-setting procedure should consider adopting minimum success 
standards (such as “at least one agency in at least three of the project countries reaches minimum 
standard ...”) enriched by a “menu” of additional, optional, success dimensions as outlined for the 
present evaluation on p. 3 (such as, “additional agencies within each country reach the minimum 
standard ....”; “the project approach is applied to other law enforcement areas ...” etc.) These 
standards and dimensions could overlap with a benchmarking approach. 


Also clarify whether the project Objective is really a separate item which is caused by the 
Outcomes or whether it is simply a summary of them (as is probably the case at present). If not, 
consider defining the Objective in terms of consequent successes in prosecutions in which the 
software and skills were used. While some agencies might be reluctant to share this information, 
not all will be (at least one agency spontaneously volunteered this information to the evaluation 
team). So the causal chain could look in outline something like this: Outputs (e.g. training, 
provision of hardware) > Outcomes (e.g. Increased use of key skills and data exchange in agency 
work) — Objective (e.g. Increased prosecutions using F23 inputs). Such a modified framework 
would need to explicitly state that “Increased prosecutions” is influenced by many factors beyond 
the control of the agencies and UNODC and cannot be directly interpreted as an indication of 
project success; nevertheless it is valuable as a statement of a final purpose of the project. 


The project logical framework is its foundational document and UNODC should provide expert 
input to ensure that it satisfies standards of evaluability and logical consistency. 


These standards and dimensions for project outcomes should then be formulated ina SMART way 
and used for real-time project monitoring, for example in two or three simple, corresponding 
spreadsheets which can be used for the Annual Reports and for day-to-day management and 
resource allocation. Standard practices for assessing training courses should be followed wherever 
possible, e.g. with brief pre-course and post-course tests of knowledge. 
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ANNEX X. MORE DETAILS ON 
RECOMMENDATION ON BENCHMARKING 
AND TRAINING STANDARDS 


This text follows on from the corresponding outline recommendation on p. 37. 


Basic training needs (“Level I’) should be covered at national level, with only minimum help from 
the project such as awareness-raising sessions and possibly provision of equipment. CARICC can 
participate in providing some standardised training modules. Basic training should cover those 
more standardised aspects which change slowly like basic skills. 


Mid-level training (“Level II”) and other inputs is already provided via CARICC using trainers 
from CARICC and the participating countries, and this should be continued. This reinforces the 
need for more advanced training for CARICC personnel, in particular on the operational side of i2. 


Mid- and advanced- level training needs to urgently address the list of emerging challenges, see p. 
30, and provide more customised inputs for individual agencies and groups of agencies. 


Advanced-level training (“Level II’’) and other inputs should be provided by trainers from beyond 
the region working in tandem with CARICC trainers who can learn through assisting in training 
provision. 


Training provision needs to link into the benchmarking minimum standards for individual skills. 
Although certificates are already issued there is the need to better quantify and evaluate training, 
for example with before and after scores on tests for each trainee, or with a trainer rating. UNODC 
should identify an external institute with expertise in constructing this kind of multi-agency 
technical training support, which can provide pedagogical input on how to construct and test 
modular minimum standards for this kind of environment. These kinds of standards can help 
provide some measure of staff progress for the agencies but also for the project. So, a minimum 
standard for an Law Enforcement Agency could specify, for example, that it should have at least 
one level III trainer and two analysts who have been trained to level II. 


Whether these benchmarks are additional to national training curricula and academies or integrated 
into them should be dealt with on a case-by-case basis. If integration is desired, the project should 
be aware that this procedure could take a considerable amount of time. 


Improved recording of staff progress in this way in collaboration with the agencies could help 
circumvent the current problem that beginner trainees are sometimes incorrectly sent to advanced 


training. 


However, CARICC still needs to demonstrate its ability to maintain its role as a centre of excellence 
and cannot be allowed to fall behind on constantly developing challenges. It must continue to 
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contribute to F23 areas of work such as identifying minimum standards, maintaining lists of 
trainers, helping with curricula, identifying new models of work etc. 
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ANNEX XI. — Expenditure breakdown 


201,360.00 ] 
23,100.00 
125,155.46 
287,067.12 
54,120.27 
84,700.97 
2,087.01 
60,026.99 
15,971.75 
66,242.09 
23,600.00 
943,431.66 
5,506.86 
120,475.09 
1,069,413.61 


International expert 
ST consultants 

Travel 

Other Personnel Costs 
Training 
Conferences/Meetings 
Exp. Equipment 
Non-exp Equip 

Rental for Premises 
General Operating Exp 


Evaluation 


Sub total 
CR 


PSC, 13% 
TOTAL 


Pn 


91,050.58 
77,819.29 
12,140.26 

1,055.59 
21,312.01 
31,342.94 


FT30_CLASS_ 160: Travel 

FT30_ CLASS 010: Staff Personnel 
FT30_CLASS_120: Contract Service 
FT30_ CLASS _ 130: Suppl Com Mater 
FT30_ CLASS _ 125: Operat Oth Costs 
FT30_ CLASS _ 135 : Equip Veh Furnit 
FT30_CLASS_155 : UN-PSC 29,061.11 


TOTAL: 263,781.78 


As the accounting system changed during this period, the headings for the two different periods are 
not fully compatible with one another. 

ANNEX. Anti-narcotics provision in National development Strategies 

In the evaluated countries covered within this project, the following relevant documents have been 
published: 
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Kazakhstan 


"Sectoral Program to Combat Drug Addiction and Drug Business in The Republic of Kazakhstan 
for 2012-2016. " Lost force by the Resolution of the Government of the Republic of Kazakhstan 
dated 29.06.2016 No. 383, so it will not be considered here. 


Kyrgyzstan 


"Anti-Drug Program of the Government of the Kyrgyz Republic". Approved by the Decree of the 
Government of the Kyrgyz Republic dated January 27, 2014 No. 54. Still active. 


Tajikistan 


"National Strategy for Combating Illicit Trafficking in Drugs for 2013-2020" Approved by the 
Decree of the President of the Republic of Tajikistan on February 13, 2013, No. 1409. Still active. 


Also, the "Sectoral program for the implementation of the National Strategy for Combating with 
illegal drug trafficking in the Republic of Tajikistan for 2013-2020 ". Approved by Decree of the 
Government of the Republic of Tajikistan of April 28, 2017 No. 211. Still active. 


Uzbekistan 


"Program of comprehensive measures to counteract abuse of drugs and their illicit trafficking for 
2016-2020". Still active. 


Summary 


Within the framework of these strategies and programs, there are tasks covering the formation in 
each country of a uniform system for the reduction of the supply and demand for drugs and the 
strengthening of trafficking within and between countries. 


In practically in all the specified program documents there are tasks related to the collection and 
analysis of operational information. So, in section 4.3.3. "Fighting against illicit trafficking in 
narcotic drugs, psychotropic substances and precursors", the programs of the Kyrgyz Republic 
sets the following tasks: 


- "Development of methods for the collection and use of operational data and evidence, including 
those authorized by courts, methods of evidence collection such as electronic surveillance and 
controlled deliveries " 


- "Creation of information-analytical systems, improvement of collection, processing and data 


analysis, the creation of electronic databases and access to them specialists, as well as the exchange 
of operational data with securing non-disclosure and protection of information sources ". 
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In the Program of Uzbekistan®®, the following is noted: 


- "In order to improve the mechanism of information exchange and organization of its centralized 
processing, a unified database format for basis of modern information-analytical software IBM i2, 
provided in the framework of international projects. Installation of this program in the analytical 
departments of law enforcement bodies of the republic, and a simplified form of the data bank was 
developed. "... .." At present work on the development of the National Database System (by analogy 
with law enforcement systems of the EU countries) ". 


The Sector Program (Action Plan) in Tajikistan has the following tasks: 


- "To improve the database of law enforcement agencies in relation to legal entities and individuals 
involved in trafficking in narcotic drugs, psychotropic substances and their precursors "; 


- "Provision of law enforcement bodies whose activities are related to disclosure of drug crimes, 


modern technical equipment and equipment, introduction of new methods of gathering information 
for disclosure of crimes ". 


60 (according to the website www. freecity.lv, unfortunately, the text is not otherwise available) 
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